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Abstract 

For  the  Air  Force  to  continue  developing  and  acquiring  today’s 
highly  complex  and  costly  weapon  systems,  it  will  have  to  depend  more 
heavily  on  computer  simulations  to  fill  in  where  hardware  is  not  avail¬ 
able  and  actual  test  ing  cannot  be  performed.  The  usefulness  of  these 
computer  simulations  are  limited,  however,  by  the  degree  to  which  the 
results  are  accepted  by  managers.  From  a  review  of  the  literature, 
however,  it  would  appear  that  no  previous  research  h  is  been  performed  t 
determine  Air  Force  program  managers'  perceptions  of  the  acceptability 
ot  computer  simulation  results.  [n  light  of  that  finding,  the  two 
objectives  of  this  thesis  were  to  identify  the  factors  affecting  the 
acceptance  of  computer  simulation  results  by  senior  program  mangers 
within  AFSC.'/ASD  and  to  determine  the  relative  importance  of  the  identi¬ 
fied  factors. 

2c '  ’  .  this  res...  h  vac  c::r!  -r^'ory  ip  nature,  the  Delphi  data 
collection  technique  was  used.  .An  expert  panel  of  nine  Air  Force 
colonels  completed  the  Delphi  survey.  The  experts  arrived  at  a  list,  of 
sixteen  factors  that  affect  die  acceptance  nf  oiimilat  results.  Of 
these  sixteen  factors,  only  ten  were  said  to  have  a  "strong  effect." 

The  conclusions  drawn  from  the  expert  peine  1  responses  show  that 
senior  program  managers  are  principally  concerned  about  the  simulation 
represent  ing  the  "real  world"  and  about  the  technical  information  incur 
pi  >rated  into  the  model.  The1  experts  also  expressed  that  the  manager 
needs  to  know  if  the  end  user  of  the  weapon  system  believes  the  simula¬ 
tion  to  be  accurate.  Several  of  the  other  factors  included: 


confidence  in  the  individual  performing  the  simulation,  ability  to 
assimilate  ,an\  recommendations,  acceptability  of  the  model  by  the  ojnmu- 
nity  that  general ly  conduces  simulations  in  the  specialty  area,  confi¬ 
dence  in  the  individual  presenting  the  results,  previous  experience  with 
simulations,  superior’s  acceptance  of  simulation  results,  and  managers' 
own  insights  and  intuit  ion. 


DETEHM  l  NAT  I  ON  OF  THE  FACTORS  AFFECT I NG  THE  ACCEPTANCE 
OF  COMPUTER  SIMULATION  RESULTS 


I.  Introduce ion 

Ck.‘nerai  Issue 

Within  \ir  Force  Systems  Command  ( AFSC) .  program  managers  arc 
responsible  fur  the  acquis  it  ion  and  test  ing  of  many  tie',  weapon  systems 
Or  tfie  US  \ir  Force  ( USaF ) ,  such  ns  the  Advanced  Cruise  Missile  (ACM), 
•he  Vi\  aneed  Techno  1  ogy  Bomber  (B-2) ,  the  Short  Range  Attack  Missile  II 
i SRAM  II)  and  others.  The  operational  commands  and  the  Air  Force  Opera¬ 
tional  Test,  and  Evaluat  i on  Center  (AFOTEC)  are  responsible  for  confirm¬ 
ing  ‘hat  the  using  command’s  needs  are  truly  being  met  by  the  systems 
being  acquired.  Ideally,  the  evaluation  of  the  systems  would  be  per- 
T'C-mod  through  the  use  of  system  test  data  only.  However,  due  to  cost  , 

‘ i me ,  safety  and  other  constraints,  this  cannot  be  done.  To  cover  all 
areas  of  the  eva’uation,  computer  simulation  is  used  to  supplement 
•a  s  t  em  test  ing.  The  value  of  these  computer  simulations  is  dependent  on 
manager*  accept  anee  of  the  simulation  results. 

Pr  ■)  •  1  em  St  at  ement 

In  this  time  of  decreasing  military  budget s  and  increasing  weapon 
•  ’  ■  m  ■  mp  b  %ity/eost,  the  military  needs  to  use  all  of  its  resources 
In  t  he  m<  st  e  f f i (  jent  .ay  possible.  If  hardware  testing  was  the  on  1 y 
method  -.f  determining  a  system’s  capabilities  and  dependability,  this 
nation’.;  war  fighting  capability  could  he  greatly  reduced  due  to  insuf- 
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ficient  ,  inaccurate  or  misleading  system  data,  This  problem  of  uncer¬ 
tain  system  infurmat ion  can  be  partially  avoided  through  the  use  of 
simulation  during  veapon  system  acquisition.  Simulation  can  be  used  in 
the  acquisition  of  major  veapon  systems  to  assist  in  design,  test  and 
managerial  decision  making.  The  usefulness  of  the  simulation  results 
are  limited,  hovever,  by  their  accuracy  and  their  acceptance  by  manage¬ 
ment.  It  is  the  area  of  acceptance  of  computer  simulation  results  by 
program  management  that  is  addressed  in  this  research. 

A  review  of  the  literature  indicates  that  no  attempt  has  been  made 
to  measure  program  manager’s  current  attitudes  tovard  the  use  of 
computer  simulations  nor  has  the  significance  of  various  factors  <-n  t  he 
usefulness  of  simulation  results  used  by  Air  Force  program  managers  been 
measured.  In  fact,  it  appears  that  no  attempt  has  been  made  to  deter¬ 
mine  the  factors  that,  affect  the  acceptance  of  computer  simulation 
results  by  program  managers . 

Oice  the  factors  affecting  the  acceptance  of  computer  simulation 
results  have  been  characterized,  then  data  can  be  collected  which  will 
shov  the  current  le\el  of  acceptance  of  simulation  results  by  program 
managers.  This  current  level  of  acceptance  data  can  then  be  used  to 
pinpoint  areas  needing  attention  so  that  the  acceptance  of  simulation 
results  can  be  improved. 

Research  ()b  jec t  j  vos 

The  main  objective  of  this  study  was  to  identify  the  factors  that 
affect  the  acceptance  of  computer  simulation  results.  A  second  objec¬ 
tive  was  to  determine  the  relative  importance  of  these  factors  to  the 


program  managers . 


Research  t/ues *  ions 

To  meet  the  research  objectives,  the  following  questions  were  posed: 

1.  What  are  the  factors  that  affect  the  acceptance  of  computer 
simulation  resuits  used  by  senior  program  managers  within  AFSC/ASD? 

2.  What  is  fhe  relative  importance  of  the  factors  identified  to 
the  senior  program  managers  within  AFSC/ASD? 

Scope 

This  research  was  limited  to  senior  AFSC  program  managers  within 
VSD.  For  the  purpose  of  this  research,  the  senior  program  manager  was 
defined  as  an  Air  Force  officer  of  the  grade  0-6  or  above  who  is  or  has 
been  the  program  manager  of  an  Air  Force  weapon  system  acquisition 
effort .  These  senior  program  managers  possess  a  Duty  Air  Force  Spe¬ 
cialty  Code  of  0002,  0016  or  0029.  This  classification  was  used  so  that 
only  those  Individuals  who  had  been  determined  to  be  "top  managers", 
experts,  would  be  surveyed  as  part  of  this  research. 

The  determination  of  the  factors  affecting  the  acceptance  of 
computer  simulation  results  was  based  on  the  purposive  judgment  of 
expert  senior  Air  Force  program  managers.  It  is  possible  that  a  survey 
of  a  different  group  of  experts  ould  result  in  a  different  set  of 
fact  ors . 

Background 

The  use  of  simulation  in  program  development  began  to  be  noticed 
during  World  War  II.  .John  Von  Neumann  and  Stanislav  Ulam  from  Los 
Alamos  Scientific  Laboratory  used  Monte  Carlo  simulations  to  solve 
neutron  diffusion  problems.  In  the  196f)’s,  computer  modeling'  and  simu¬ 
lation  began  to  be  included  in  college  and  liniversity  courses  (16:1). 


In  the  past  several  years,  computer  simulation  has  experienced  extensive 
growth.  In  fact,  in  March  1989  the  Department  of  Defense  (DOD) 
presented  the  Critical  Technologies  Plan  to  Congress,  that  identifies 
the  22  most  essential  technologies  to  the  "long  term  qualitative 
superiority  of  US  weapon  systems"  (15:25).  This  plan  presented  simula¬ 
tion  and  modeling  as  number  six.  Computer  simulation  has  several  advan¬ 
tages  over  real  experiments.  Simulations  cost  less;  they  only  take 
seconds  to  execute  instead  of  weeks  or  years;  exact  replication  can  be 
achieved;  and  experiments  can  be  performed  that  would  otherwise  be  too 
dangerous  (18:8). 

Computer  simulation  normally  involves  three  main  steps.  The  first 
is  to  characterize  the  system  to  be  modeled  or  "building  the  theory". 

In  this  step  the  inputs  and  the  outcomes  must  be  determined,  the  inter¬ 
nal  operation  of  the  sub-systems  must  be  known  and  any  limitaticis  of 
the  system  must  be  established  (21:174). 

The  second  step  is  the  trans format  ion  of  the  system  into  a  model 
(21:174).  One  of  the  most  common  methods  of  doing  this  is  through  the 
use  of  flow  charts.  Flow  charts  allow  for  a  smooth  logical  order  of 
events  to  be  described  easily  and  allows  for  easier  verification  of 
software  coding  later  in  the  process. 

The  third  step  is  the  conversion  of  the  flow  chart  (model)  into  a 
computer  program.  Usually  a  higher  level  programming  language,  such  as 
FORTRAN,  or  a  specialized  simulation  language,  such  as  SLAM  II,  is  used 
(21:171). 

In  addition  to  creating  the  computer  program,  the  model  must  be 
proved  to  be  a  valid  representation  of  the  original  system.  The  process 
of  determining  validity  takes  place  in  all  three  steps:  the  characteri- 
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zat ion,  the  flow  chart,  and  the  computer  programming  (21:178).  Care 
must  tie  taken  not  to  assume  the  final  program  is  valid  on  the  basis  of 
only  validating  the  first  two  steps,  as  Glazer  says  "findings  from  such 
simulations,  whi le  internally  valid,  may  not  be  totally  externally  valid 
due  to  the  multitude  of  complicating  factors  present  in  the  real  world" 
(0:5.6)  .  Robert  Sargent  states:  "To  obtain  a  high  degree  of  confidence 
in  a  model  and  its  results,  comparison  of  the  model’s  and  system’s 
input -out  put  behavior  for  at  least  two  different  sets  of  experiment, 
conditions  is  usually  required"  (20:36).  The  importance  of  validating 
the  model  cannot  be  assumed  to  be  of  little  importance;  it  is  one  of  the 
most  important  actions  that  must  be  taken  before  a  simulation  can  be 
declared  of  any  value. 

The  modeling  and  programming  steps  are  only  preparation  for  the  true 
simulation  effort.  Once  the  model  has  been  validated,  it  can  then  be 
used  to  conduct  "trial -and-error  experiments"  (12:56;  18:5).  Tvo  exam¬ 
ples  of  these  experiments  are  hardware  system  development  and  war 
gaming.  In  these  examples,  the  user  of  the  model  is  able  to  perform 
simulations  using  a  variety  of  system  characteristics.  With  each 
successive  simulation,  the  user  is  able  to  refine  the  system  character¬ 
istics  allowing  progress  to  be  made  toward  a  predetermined  end  goal. 

Once  the  predetermined  end  goal  is  reached,  the  system  characteristics 
can  be  passed  to  the  next  stage  of  development.  In  the  case  of  hardware 
system  development,  the  next  stage  might  be  hardware  fabrication  or  the 
next  stage  could  be  an  integrated  war  plan  in  the  case  of  a  war  gaming 
simulation  (12:56;  18:5). 

Simulation  has  been  very  important  in  the  development  of  new  hard¬ 
ware  systems.  One  good  example  is  the  use  of  Computer-Aided  Brgineering 


(CAE),  which  has  increased  flexibility  in  design  for  both  the  managers 
and  the  engineers.  The  managers  are  able  to  monitor  progress  of  the 
system  with  a  high  degree  of  accuracy  and  the  engineer  can  "stress  and 
test"  the  design  throughout  the  design  process  (1:13,42). 

War  gaming  is  a  simulation  in  which  models  from  multiple  offensive 
and/or  defensive  weapon  systems  are  integrated  with  a  model  that  charac¬ 
terizes  a  wartime  environment.  The  .Army  is  using  ARTBASS,  a  battlefield 
simulation,  to  train  battalion  commanders  (12:58).  The  Air  Force  is 
developing  a  network  of  computers  that  will  simulate  the  Strategic 
Defense  Initiative  (SDI)  command,  control  and  communication  system  reac¬ 
tion  to  a  nuclear  attack  (2:18). 

All  of  the  efforts  outlined  in  the  description  above  is  fine  in  and 
of  itself;  however,  this  information  does  not  address  how  wi 1 1 ing 
management  is  to  truly  accept  and  use  these  computer  simulations.  It 
really  does  not  matter  how  much  effort  goes  into  the  development  and 
validation  of  a  simulation  if  management  is  not  willing  to  use  it.  An 
exploration  of  the  literature  addressing  this  issue  of  managerial 
acceptance  of  computer  simulation  results  indicates  little  or  no 
research  has  been  done  in  this  area. 

Robert  Sargent  points  out  that  the  primary  factor  in  determining  an 
individual’s  confidence  in  simulation  is  the  degree  of  validation  that 
has  been  performed.  Sargent  also  says  that  verification  of  correct 
software  coding  affects  the  confidence  in  simulation  (28:33,38). 

William  Morris,  in  his  'writings  on  managerial  acceptance  of  industrial 
engineering  efforts,  agrees  that  scientific  validity  is  one  of  the  best 
criteria  to  use  for  the  acceptance  of  recommendat ions .  However,  he  goes 
further  to  suggest  that  there  are  a  number  of  other  factors  that  are 
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considered.  Morris  states  that  managers  also  depend  on:  personal 
insights  and  intuition,  an  understanding  of  the  technical  aspects  of  the 
recommendation,  acceptance  of  recommendations  by  the  manager’s  supe¬ 
riors,  acceptance  of  equivalent  recommendations  by  manager's  peers,  the 
degree  to  which  the  manager  is  able  to  assimilate  recommendations, 
manager’s  confidence  in  the  individual  presenting  the  recommendations, 
manager’s  confidence  in  the  individual  performing  the  study,  the  degree 
(magnitude)  of  effort  used  to  develop  the  recommendations,  cost  of 
obtaining  recommendations,  the  manager’s  previous  experience  with  that 
type  of  study,  and  the  skill  used  in  presenting  the  recommendat ions  to 
the  manager  (14:16-19).  In  addition  to  these  factors,  there  is  some 
reason  to  believe  that  formal  education  courses  in  computer  simulation 

could  have  some  effect  on  the  acceptance  of  their  results  (7). 

> 

Key  Terms  and  Def ini t ions 

Computer  simulation  is  a  process  of  using  a  mathemat i cal /logical 
representation,  model,  of  some  real  world  system  that  is  manipulated  to 
perform  near  realistic  experiments  on  a  computer  (3:7;  16:6;  18:5;  19:6; 
21 :  173-174) . 

Verification  is  the  process  of  confirming  that  the  computer  software 
that  comprises  the  model  executes  as  intended.  In  verification,  there 
is  no  attempt  to  compare  simulation  results  to  the  system  being  modeled. 
(3:7,13;  16:67:  20:33,35) 

Validation  is  the  process  of  comparing  the  results  of  the  computer 
simulation  to  the  real  world  system  to  determine  how  well  the  real  world 
system  is  replicated  (3:7,13;  16:67;  18:9;  20:33).  There  are  numerous 
types  of  validation.  Some  of  the  most  common  types  are:  expert  opinion, 
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comparison  to  other  models,  event  validity,  extreme-condition,  face 
validity  and  historical  methods. 

An  expert  is  an  individual  who  has  extensive  experience  and  is 
prominent  in  an  area  of  study. 


Overview 

This  chapter  outlined  the  Air  Force’s  need  for  simulation  and  a 
brief  history  of  computer  simulation.  The  problem  to  be  addressed,  the 
research  objectives  and  the  research  questions  were  presented  as  well  as 
a  review  of  the  related  literature.  As  part  of  the  literature,  a  list 
of  factors,  presented  by  William  Morris,  affecting  the  acceptance  of 
industrial  engineering  recommendations  was  outlined. 

The  following  chapters  describe  the  research  in  more  detail. 

Chapter  II  presents  the  research  plan  for  obtaining  the  desired  data 
from  senior  Air  Force  program  managers,  while  Chapter  III  covers  the 
research  findings  and  data  analysis  performed.  In  Chapter  IV,  the 
research  is  summarized  and  recommendations  for  future  research  and 
application  of  this  research  are  outlined. 
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II.  Me t hodo 1 01 


During  the  review  of  literature,  no  previous  research  was  found 
which  addressed  the  weapon  system  program  manager’s  perceptions  of 
computer  simulation  results.  However,  Wi 1 1 iam  Morris  did  discuss  fac¬ 
tors  affecting  the  acceptance  of  industrial  engineering  work  (14:16-18). 
As  this  is  the  first  attempt  to  evaluate  a  program  manager’s  perceptions 
of  the  utility  of  computer  simulation  results  in  weapon  system  acquisi¬ 
tion  decisions,  an  exploration  approach  to  research  was  used  (8:62). 

This  exploratory  research  focused  on  the  determination  of  factors 
affecting  the  acceptance  of  computer  simulation  results  by  senior  Air 
Force  acquisition  managers.  The  Delphi  data  collection  technique  was 
used  to  identify  the  factors  usingMorris*  factors  as  the  foundation  to 
begin  research. 

The  factors  that  were  used  as  the  foundation  in  this  evaluation 
were:  understanding  of  the  technical  information  used  in  simulation 
models,  degree  to  which  a  computer  model  represents  the  "real  world", 
verification  of  correct  software  coding,  agreement  with  manager’s  in¬ 
sights  and  intuition,  acceptance  of  simulation  results  by  superiors, 
acceptance  by  other  program  managers,  degree  (magnitude)  of  effort  used 
in  developing  the  s imulat ion  model ,  manager’s  ability  to  understand 
results,  manager’s  confidence  in  the  individual  presenting  the  results, 
confidence  in  the  individual  performing  the  simulation  study,  cost  of 
obtaining  simulation  results,  manager’s  previous  experience  with  simula¬ 
tion  studies,  skill  used  in  presenting  the  simulation  results,  and 
formal  education  courses  in  computer  simulation  design  and  use. 
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The  Delphi  technique  is  simply  a  set  of  procedures  "used  for  the 

elicitation  of  opinions  with  the  objective  of  obtaining  a  group  response 

of  a  panel  of  experts"  (4:3;  5:v).  This  technique  is  a  qualitative 

method  that  places  the  emphasis  on  informed  judgment.  The  technique  is 

based  on  the  oid  adage  "Two  heads  are  better  than  one,"  especially  in  a 

situation  where  exact  knowledge  is  not  available.  Norman  C.  Dalkey,  one 

of  the  original  developers  of  the  Delphi  technique,  said  that 

the  Delphi  procedure  is  one  of  the  most  efficient  I  know 
for  "uncover ing"  the  implicit  models  that  lie  behind  opinions 
in  the  "soft"  areas.  One  of  the  most  valuable  side-products  of 
a  Delphi  exercise  concerned  with  strategic  bombing  was  the 
skeleton  of  a  model  which  was  later  fleshed  out  in  great 
detail.  (6:9) 

Traditionally,  group  decision  making  has  resulted  in  more  accurate 
conclusions  than  individual  decisions.  This  is  a  result  of  having  a 
greater  amount  of  information  available  because  of  having  more  individ¬ 
uals’  experiences  and  that  more  complex  problems  can  be  addressed 
because  there  are  more  individuals  to  consider  the  problem.  However, 
there  has  been  several  major  disadvantages  identified  in  connection  with 
face-to-face  panels  or  committees.  First,  there  is  at  least  as  much 
misinformation  in  a  group  as  there  is  for  an  individual.  Second,  the 
existence  of  strong  peer  pressure  to  agree  with  the  majority.  Next,  the 
goal  of  the  group  changes  from  its  primary  intent  to  one  of  finding  an 
answer  that  will  not,  offend  anyone,  even  though  no  individual  member  may 
agree  strongly  either.  Another  problem  is  that  the  minority  opinion  may 
overpower  the  rest  of  the  group  by  its  strong  vocalization  and  the 
repetition  of  the  arguments.  Face-to-face  panels  also  have  a  problem 
with  being  vulnerable  to  the  influence  of  "dominant  indl  I  lu^ls." 


sixth  disadvantage  is  that  individual  members  of  the  panel  may  have 
vested  interests  in  the  outcomes  of  the  discussions  which  may  cause  the 
individual  to  ignore  all  facts  and  logic,  and  concentrate  solely  on 
winning  the  argument.  Finally,  the  entire  panel  may  share  a  common 
bias.  This  common  bias  is  counter  to  the  intent  of  this  technique  which 
is  to  have  individuals  with  different  biases  causing  a  canceling  effect. 
(12:15-16).  The  Delphi  technique  has  three  features  that  attempt  to 
counter  the  negative  effects  of  face  to  face  panel  discussion:  anonym¬ 
ity,  iteration  with  controlled  feedback,  and  statistical  group  response 
(5:16;  13:16). 

The  first  feature,  anonymity,  is  achieved  through  the  use  of  ques¬ 
tionnaires  or  computer  communication.  This  feature  reduces  the  effect 
of  dominant  individuals.  The  iteration  with  controlled  feedback  feature 
allows  the  group  to  move  toward  consensus  through  iterations,  rounds,  of 
expressed  opinions  with  a  group  moderator  that  assures  that  only  rele¬ 
vant  information  is  made  available  through  the  feedback  process.  Sta¬ 
tistical  group  response  generally  includes  a  presentation  of  the  mean 
and  median  of  the  panel’s  opinion  and  the  degree  of  spread  from  the 
mean.  This  feature  assures  that  all  opinions  are  represented  in  the 
response,  not  just  the  majority  opinion.  These  features  have,  in  fact, 
been  found  to  correct  for  many  of  the  problems  connected  with  collecting 
expert  opinion  in  a  classical  panel  or  round-table  forum  (5:16;  13:16- 
17)  . 

Another  advantage  of  the  Delphi  technique  over  face-to-face  discus¬ 
sions  is  that  it  is  more  accurate.  Experiments  performed  by  both  R. 
Campbell  and  Norman  Dalkey  show  that  information  resulting  from  the  use 
of  the  Delphi  technique  are  more  likely  to  be  correct  than  those  result- 
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ing  f r  ’  a  face-to-face  discussion.  Campbell’s  experiment  involved  the 
use  of  a  group  of  graduate  students  vho  were  asked  to  make  short-range 
economic  forecasts  from  a  set  of  supplied  data.  The  students  were 
divided  into  groups  that  used  the  Delphi  and  those  vho  used  any  type  of 
"free  communicat ion"  desired.  The  result  of  the  experiment  vas  that  the 
Delphi  forecasts  were  more  accurate  in  13  cases,  after  the  fourth  round 
and  the  non-Delphi  groups  were  more  accurate  in  only  two  cases. 

Dal  key  used  a  group  of  ten  graduate  students  vho  were  divided  into 
tvo  groups'.  In  this  experiment,  the  face-to-face  discussion  group  vas 
given  specific  instructions  as  to  the  procedures  that  were  to  be  used. 
Toe  Delphi  group  used  the  Delphi  procedures.  The  result  of  Dalkey’s 
experiment  was  that  the  Delphi  group  vas  more  accurate  in  13  cases  and 
the  face-to-face  group  vas  more  accurate  in  7  cases  (4:7-12;  5:21-23). 
Oie  author  stated: 

Dalkey’s  findings  show  that  the  group  process  in  Delphi  does  an 
efficient  job  of  extracting  information  from  a  panel  as 
compared  with  face-to-face  interaction.  While  none  of  these 
findings  can  "validate"  a  Delphi  forecast,  taken  together  they 
indicate  that  when  it  is  necessary  to  use  expert  opinion, 

Delphi  is  a  good  way  of  getting  it.  (13:23-24) 

The  Delphi  technique  has  not  been  developed  without  its  critics.  H. 
Sackman  charged  that  its  biggest  problem  is  that  the  results  of  a  Delphi 
exercise  are  not  verifiable.  He  says  this  is  a  major  violation  of  the 
rules  of  scientific  research.  Sackman  also  had  other  criticisms  includ¬ 
ing:  lack  of  documentation  of  experience  and  qualifications  of  the 
experts,  biased  results  due  to  respondent  dropout,  the  time  required  of 
the  participants  to  complete  all  rounds,  and  encourages  "snap  judgments" 
( sackman: 13-22) .  Most  of  the  Delphi  studies  that  Sackman  used  to 
develop  these  criticisms  were  of  the  forecasting  type.  Since  this 
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research  does  not  involve  forecasting  and  focuses  on  the  determination 
of  factors,  these  criticisms  will  have  little  or  no  impact  on  the  con¬ 
clusions  drawn  from  this  use  of  the  Delphi  technique. 

The  first  step  in  implementing  the  Delphi  technique  is  the  selecting 
of  panel  members.  This  step  is  probably  one  of  the  most  difficult  and 
most  critical  portions  of  the  Delphi  technique.  The  first  problem  is 
defining  what  an  expert  is  in  the  technical  area  of  research.  Then  the 
next  problem  is  finding'  individuals  who  are  "true"  experts  in  that 
field.  Another  problem  is  selecting  a  panel  from  the  available  experts 
that  will  be  unbiased  and  represent  a  cross  section  of  available  thought 
on  the  subject,  being  addressed  (  1:3-1:  13:26-29).  Martino  says  "It 
cannot  be  emphasized  too  strongly  that  choosing  the  panel  is  the  most 
important  decision  the  panel  moderator  will  make,  and  considerable 
effort  in  making  a  good  selection  is  fully  justified"  (13:29).  Authors 
Jones  and  Twiss  recommended  that  the  number  of  experts  included  on  the 
panel  should  be  between  18  and  50.  It  was  pointed  out  that  less  than  10 
panel  members  may  limit  the  availability  of  varied  backgrounds  and  the 
accuracy  of  results  (11:229). 

Once  the  questionnaire  has  been  developed  and  pretested  for  accuracy 
and  validation  of  statements,  the  first,  round  can  be  initiated.  Accom¬ 
panying  the  first  round  should  be  a  clear  statement  of  the  Delphi  pro¬ 
cedure.  After  the  responses  are  returned,  medians  should  be  calculated 
for  each  fjuest ion  and  any  comments  that  were  returned  should  be  reviewed 
for  relevance.  The  medians  and  relevant  comments  are  then  incorporated 
into  the  survey  and  returned  to  the  panel  members.  In  round  two,  the 
panel  members  are  asked  to  consider  their  responses  in  relation  to  the 
mod : an  response  and  the  comments  provided,  and  make  any  changes  and 
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comments  that  they  feel  are  appropriate.  This  process  of  rounds  is 
continued  until  consensus  is  reached  or  stability  is  achieved  on  at 
least  79  percent  of  the  questions  with  a  minimum  of  two  rounds  being 
executed.  Consensus  is  defined  as  at  least  59  percent  agreement  within 
the  panel  and  stability  is  achieved  when  no  further  change  in  opinion  is 
expressed.  Once  consensus  or  stability  is  observed,  the  opinions 
expressed  in  the  final  round  are  regarded  as  the  panel's  conclusions 

i  1:  1-1;  !*-20>  . 

Mean  ;uid  median  calcuiat ions  are  used  in  the  analysis  of  the  survey 
results.  These  calculat  ions  are  used  as  a  measure  of  central  t endeney 
T  r  the  da: a.  "The  median  of  a  set  of  measurements  is  defined  to  be  the 
mi du  1  e  value  ..hen  the  measurements  are  arranged  from  lowest  to  highest" 
(17:33).  The-  median  of  an  odd  number  of  measurements  is  simply  the 
middle  value.  The  median  of  an  even  number  of  measurements,  however,  is 
the  average  of  the  two  middle  values.  "The  arithmetic  mean,  or  mean,  of 
a  set  if  measurements  is  defined  to  be  the  sum  of  the  measurement s 
divided  by  the  total  number  of  measurements"  (17:35). 

Populat ion 

Since  the  focus  of  this  research  is  the  determination  of  factors 
affect  ing  the  acceptance  of  computer  simulation  results  by  weapon  system 
program  managers,  an  expert  must  be  experienced  in  the  use  of  simulation 
results  in  dei  i s i on  making.  In  addition  to  decision  making,  the 
individual  must  also  be  in  a  position  of  responsibility  requiring  a 
determination  of  acceptance  of  the  simulation  results.  Within  the  Air 
Force,  a  position  having  these  characteristics  is  that  of  program  man- 
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agvr.  The  ref.  >ro,  the  sample  of  experts  vas  drawn  from  a  population  of 
;  r  gr.iin  miuia^vr^  in  t  he  acquis  i  t  ion  of  weapon  systems  for  the  I'SAF. 

S.-unp  !J  na 

F  r  the  purposes  of  this  research,  it  was  decided  that  experts  in 
hi.  isi'iti  mat. mg',  as  it  applies  to  Air  Force  weapon  system  acquis  i  t  ion, 

■m  he  defined  as  officers  of  the  grade  0-6  or  above  with  a  Duty  Air 
F-  e  s|  i  c  i  a  1  t  y  (ode  )  DAFSC  >  of  0002,  0010  or  0029.  These  DAFSf's  are 

:  ’  it  •  ir.d  i  v  Final  s  who  have  t  lie  experience  ;ind  knowledge 

•  :  •  lire"’  ’  h«  •  \i  r  Force’ s  trios'  import  ant  programs .  It  -as 

:•  -  o  i*  :  •  ira-  ;xdu •  1  members  non-r.-indomly  from  AFSC'/ASD  booaus* •  ■  ;< f  the 

;u,F  .  ,,  tire!  :u‘.d  !  heir  pr-  •:<  imi  ty  t  o  \FIT.  It  -as  also  assume'! 

•  ha'  ,  as  a  -h  he,  *  hose  perseniie'  -ere  i a larac  t  e r  i  s  t  j  <•  ,..f  t  he  i  r  peers 
avs  :  grind  ’hr-  >ugh.  .ut  lie  \ir  Ft  >rce  acquis  i  ‘  ion  community.  A  list  of  25 
individual-  possessing  the  necessary  charac t or i s t i cs  was  received  from 
'  Fa  perst  .nr  it  1  ■ .f  f  lee  at  \SD.  Fr«  >m  the  25  individuals  listed  20  0-6s 
sere  s.  lifted  '  par*  icipate  in  the  research.  [t  should  be  noted  that 
during  t  hi  •  execution  >f  this  research  one  of  the  panel  members  was 

:  r  mett  ii  •  Brigadier  General  . 

I  it  Iph. ;  s  .;-v.  a  In  •>.  t  _j_  j  ment 

n.<  survey  was  developed  using  the  factors  outlined  by  Morris  and 
-as  -  rd-d  >  '  t  lia*  a  five  point  I.ikert  scale  could  be  used.  Ample  space 
at  -  ;  !*•  v  ided  f.  -r  ■  inmenf  s  . 

F’ret  ist  \  ■  .  imp’ !  t -t  er)  survey  was  p»re  tested  by  four  members  of  the 
\F  I T  Selii  if  >  1  iif  Systems  ;uul  I.ogist  it  rtuuni  t1  faculty.  Of  these  four 

members ,  .  .no  is  an  instructor  of  "Research  Methods"  and  is  very-  knowl¬ 
edgeable  in  the  Delphi  technique,  one  hits  used  the  Delphi  technique  in 
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previous  research  and  has  worked  within  a  SPO  organization,  the  third  is 
experienced  in  the  use  of  computer  simulations  and  the  interpretation  of 
their  results,  and  the  fourth  is  familiar  with  general  SPO  operations 
and  administ  rat,  i ve  procedures.  Comments  provided  by  these  individuals 
were  incorporated  into  the  survey  before  the  first  round  was  sent  out. 
dhe  individuals  who  took  part  in  the  pretest  did  not  participate  in  the 
actual  Delphi  exercise. 

K  .und  i  »no  The  first,  round  Delphi  survey  included  :}  demograph i ■' 
questions,  lb  questions  pertaining  to  the  factors  outlined  by  Morris, 
and  an  open  ended  request  for  comments  and  additional  factors  that 
should  be  considered.  The  first  round  Delphi  survey  is  displayed  in 
Appendix  A.  Of  the  8  demographic  questions,  quest  ions  7  and  8  vu-r* 
designed  to  eliminate  from  the  exercise  those  individuals  that  had  tv>t 
had  any  exposure  to  computer  simulation  studies  or  the  use  of  simulation 
results.  Questions  10  through  28  used  a  five  point  Likert  scale  to 
measure  the  expert’s  responses.  Each  expert  was  asked  to  respend  b\ 
selecting  the  descriptive  term  that  most  accurately  reflected  their 
judgment  on  each  question.  The  Likert  scale  used  is  depicted  in  Figure 


VERY  STRONG  STRONG  MODERATE  LITTLE  NO  NO  OPINION/ 

LFFLCT  EFFECT  EFFFXT  EFFECT  EFFECT  DON’T  l  NT >KRST AM  > 

Fig.  1.  Likert  [fat  ing  Scale 

The  Likert  scale  uas  selected  for  use  in  this  re.-><  areh  because  -  >f 
several  ad van f ages  over  other  methods.  This  scale  is  easy  and  qui-'k  t 
r-oistmrt  ,  ;md  is  easy  to  interpret  allowing  for  ma.xirmim  part  i  e  ipat  i  <  >n 
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by  panel  members.  The  Likert  scale  is  "probably  more  reliable"  and 
provides  a  greater  volume  of  data  than  other  similar  scales  (8:255-258). 

Question  24  was  designed  as  an  attempt  to  rank  order  the  14  factors 
in  order  of  importance.  A  column  vas  also  provided  labeled  "Personal 
Knowledge  Required"  in  an  attempt  to  determine  which  of  the  factors  were 
used  personally  by  the  program  managers  and  which  they  believe  were  the 
responsibility  of  a  lower  level  manager. 

The  wording  for  the  cover  letter  and  the  survey  instructions  to  the 
panel  members  were  taken  from  Captain  Ral inda  Gregor’s  1988  thesis  (10). 

The  participants  were  asked  to  return  the  surveys  in  ten  days. 
Telephone  followup  calls  were  made  after  three  weeks  to  those  panel 
members  who  had  not  responded.  At  the  end  of  the  fifth  week  only  one 
additional  survey  had  been  received.  This  resulted  in  a  total  of  16 
surveys  being  returned,  an  80  percent  return  rate.  Of  the  16  returned 
surveys ,  7  panel  members  stated  that  they  had  load  no  exposure  t.o  the  use 
>  >  f  Simula  ion  results  on  the  job.  This  yielded  9  panel  members  to  take 
part  in  the  second  round  of  the  exercise. 

Round  Two  The  second  round  Delphi  survey  included  all  factors, 
even  if  consensus  had  been  reached  on  the  first  round.  This  was  done  to 
allow  the  panel  members  an  opportunity  to  see  the  first  round  results 
and  make  any  comments  or  changes  desired.  The  panel’s  ratings  were  fed 
back  in  the  form  of  frequency  counts  for  each  descriptive  term,  as  well 
as  ;tny  comment  s  that  had  been  made  for  each  question.  Two  additional 
factors  were  suggested  as  part,  of  the  first  round.  These  two  new  fac¬ 
tors  were  included  as  new  questions  in  the  second  round  and  panel  mem¬ 
bers  were  asked  to  determine  where  within  the  prioritized  list  of  fac¬ 
tors,  round  one  question  21,  they  should  he  placed. 
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Iri  this  research,  the  tool  used  to  feed  back  an  earlier  round’s 
results  vas  a  frequency  distribution  of  the  responses,  not  the  standard 
mean,  median  and  interquartile  range  (IQR).  This  was  done  in  order  to 
provide  the  most  accurate  feedback  possible.  This  also  avoided  the 
problem  of  the  panel  members  thinking  that  a  designator  1  through  5, 
being  assigned  to  the  descriptive  terms,  was  in  some  way  showing  an 
order  of  desirability. 

The  second  round  Delphi  survey  is  displayed  in  Attachment  B.  The 
wording  for  the  second  cover  letter  and  the  survey  instructions  to  the 
panel  members  were  taken  from  Captain  Ralinda  Gregor’s  1988  thesis  (10). 

Telephone  followup  calls  were  made  after  two  weeks  to  those  panel 
members  who  had  not  responded.  At  the  end  of  the  fourth  week  one 
additional  survey  had  been  received.  This  resulted  in  a  total  of  eight 
surveys  being  returned,  an  89  percent  return  rate. 

Summary 

This  chapter  provided  a  synopsis  of  the  process  used  to  develop  a 
profile  of  factors  affecting  the  acceptance  of  computer  simulation 
results.  A  description  of  the  Delphi  data  collection  technique  was 
provided  as  well  as  an  analysis  of  the  population  and  sample.  It  was 
explained  that,  Morris’  list  of  factors  was  used  as  a  foundation  for  the 
round  one  survey  and  that  15  responses  out  of  the  28  surveys  distributed 
were  received.  Only  9  of  the  15  responses  were  used  in  the  second  round 
of  the  exercise.  The  next  chapter  presents  and  analyzes  the  data 
obtained  in  the  research. 
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III.  Findings  and  Ana lysis 

This  research  dealt  with  determining  the  factors  affecting  the 
senior  Air  Force  program  manager’s  acceptance  of  computer  simulation 
results.  Chapter  II  described  the  research  plan,  including  a  descrip¬ 
tion  of  the  Delphi  data  collection  technique.  This  chapter  presents  a 
demographic  characterization  of  the  panel  members  who  participated  in 
the  Delphi  exercise  and  the  Delphi  results. 

Delphi  Pane  1 

The  first  round  Delphi  panel  consisted  of  nine  colonels  representing 
61  years  of  combined  SPO  experience.  The  panel  also  contributed  36 
years  of  combined  computer  simulation  experience.  During  the  second 
round  of  the  Delphi  survey,  one  of  the  colonels  did  not  respond  reducing 
the  years  of  combined  SPO  experience  to  59.  All  panel  members  are 
presently  assigned  to  AFSC  with  six  of  the  participants  having  pre¬ 
viously  been  assigned  to  Strategic  Air  Command  (SAC)  or  Tactical  Air 
Command  (TAC) .  Four  of  the  officers  had  been  assigned  to  either  Head¬ 
quarters  Air  Force  (HAF),  United  States  Air  Forces  in  Europe  (USAFE)  or 
the  Pacific  Air  Forces  (PACAF).  Three  individuals  have  been  in  .AFSC  for 
their  entire  Air  Force  career. 

In  the  area  of  educational  background,  seven  of  the  members  received 
•heir  bachelor’s  degree  in  engineering,  with  the  two  remaining 
possessing  undergraduate  degrees  in  business  administration.  However, 
only  four  of  the  panel  members  earned  graduate  degrees  in  engineering' 
while  six  earned  degrees  in  either  business  administration  or  program 
management  (note  that  one  individual  had  earned  masters  degrees  in  both 
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engineering  and  business  administration).  One  panel  member  does  have  a 
doctorate  in  engineering.  Interestingly,  only  one  of  the  panel  members 
had  ever  taken  any  formal  computer  simulation  education  courses  and  even 
then,  he  only  had  taken  two  courses.  Two  of  the  nine  had  participated 
in  the  development  of  at  least  one  computer  model  and  all  had  used 
computer  simulation  results  in  decision  making. 

Delphi  Survey 

The  round  one  survey  consisted  of  24  questions,  including  nine 
demographic  (questions,  fourteen  Likert  scale  items  and  one  relative 
importance,  ranking,  question.  The  round  two  survey  contained  seventeen 
questions.  Sixteen  Likert  scale  questions,  two  of  which  had  been  added 
as  a  result  of  the  first  round,  and  one  relative  importance,  or  ranking, 
question.  The  Delphi  exercise  was  terminated  at  the  conclusion  of  the 
second  round. 

Round  One  Twenty  round  one  surveys  were  sent  out.  Of  the  20 
surveys  distributed  16  were  returned.  This  yielded  an  80  percent  return 
rate,  however  seven  individuals  disqualified  themselves  as  experts  in 
the  use  of  computer  simulation  results.  Follow-up  telephone  calls  were 
made  to  the  remaining  four  survey  recipients;  however,  due  to  temporary 
duty  assignments  (TDYs),  scheduled  leave  or  other  reasons,  these 
individuals  did  not  complete  the  survey.  A  telephone  call  was  made  to 
one  of  the  panel  members  to  clarify  his  demographic  data. 

A  summary  of  the  panel  members’  responses  to  the  first  round  Likert 
scale  questions  is  shown  in  Table  1.  Each  term  in  the  Likert  scale  was 
assigned  a  numerical  rating  with  1  being  assigned  to  "No  Effect"  and  5 
being  assigned  to  "Very  Strong  Effect."  A  rating  of  0  was  assigned  to  a 
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Table  1.  Likert  Responses  --  Round  One  Delphi  Survey 


Factor 


Personal 

Ratings  Know  1  edge 

5  4  3  2  1  9  Mean  Median  Required 


Understanding  of 

technical  information  1  6 

Degree  represents 

"real  world"  4  5 

Code  verification  1  2 

Agreement  with  personal 

ins ights/ intui t ion  3  4 

Supervisor  acceptance  1  4 

Other  program  managers 
acceptance  9  1 

Degree  (magnitude)  of 
development  effort  9  1 

Ability  to  understand 

recoiranendat  ions  from 

results  3  5 

Confidence  in  individual 
presenting  results  2  6 

Confidence  in  individual 
performing  simulation  2  5 

Cost  of  performing 
simulation  9  9 

Personal  previous 

experience  9  5 

Ski  1 1  used  in 

presenting  results  9  3 

Formal  education 

courses  9  1 


2 

9 

9 

9 

3.83 

4 

190% 

9 

9 

1 

9 

4.44 

4 

89% 

3 

3 

9 

9 

3.11 

3 

56% 

2 

9 

9 

9 

4.11 

4 

50% 

4 

9 

9 

9 

3.67 

4 

50% 

4 

4 

9 

9 

2.67 

3 

33% 

3 

3 

2 

9 

2.33 

2 

56% 

9 

9 

9 

1 

4.38 

4 

11% 

1 

9 

9 

9 

4.11 

4 

22% 

1 

1 

9 

9 

3.89 

4 

33% 

4 

3 

2 

9 

2.22 

2 

11% 

4 

9 

9 

9 

3.56 

4 

11% 

6 

9 

9 

9 

3.33 

3 

9% 

3 

5 

9 

9 

2.56 

2 

89% 
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response  of  "No  Opinion/Don’t  Understand."  This  sunmary  includes  the 
calculated  mean  and  median  for  each  factor.  It  also  includes  the  per¬ 
centage  of  respondents  that  indicated  they  required  personal  knowledge 
of  the  information  as  opposed  to  being  dependent  on  individuals  lower  in 
the  organization  for  the  knowledge.  The  last  question  of  round  one 
called  for  the  expert  to  rank  order  the  factors,  w-ith  "1"  being  the  most 
important.  The  ranking  is  shown  in  Table  2  and  includes,  for  comparison 
purposes,  the  calculated  mean  and  median  values,  from  Table  1,  for  each 
of  the  Likert  questions.  A  determination  of  consensus  was  not  made 
after  the  first  round  since  a  minimum  of  two  rounds  were  required. 
Comments  made  by  the  panel  members  are  listed  in  Appendix  C. 

Table  2  shows  that  seven  of  the  factors  clearly  had  a  "strong 
effect",  a  median  of  4,  on  the  manager.  An  eighth  factor  with  a  median 
of  "4"  was  also  identified;  however,  it  was  ranked  last  as  part  of  the 
ranking  exercise.  This  difference  brings  into  question  the  accuracy  of 
either  the  ranking  or  the  Likert  response  of  that  last  factor.  It  would 
appear  that  a  combination  of  small  sample  size,  extreme  views  and  nega¬ 
tive  connotation  of  "intuition"  being  used  by  top  management  in  decision 
making  led  to  this  factor  receiving  a  low  initial  ranking  while  still 
having  a  "strong  effect"  on  the  Likert  scale.  The  high  Likert  median 
appears  to  be  an  accurate  measure  of  this  factor’s  effect.  This  can  be 
traced  to  research  which  shows  that  the  lack  of  information,  uncertainty 
attributed  to  information,  and  conflicting  goals  of  an  organization 
force  the  manager  to  depend  on  "personal  insights  and  intuition"  to 
arrive  at  good  decisions  (22).  One  of  the  experts  said  that  "Unfortun¬ 
ately,  we  all  suspect  the  model  builder,  because  simulations  have  been 
used  for  political  purposes;  thus,  and  again  unfortunately,  simulations 
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Table  2.  Ranked  Factors  --  Round  One  Delphi  Survey 


Likert 


Rank 

Median 

Mean 

Factor 

1 

4 

4.44 

Degree  to  vdiich  model  represents  the  "real  world" 

2 

4 

3.83 

Technical  information  incorporated  in  the  computer 
model 

3 

4 

3.89 

Confidence  in  individual  performing  simulation 

4 

4 

4.11 

Confidence  in  individual  presenting  results 

5 

4 

4.38 

Ability  to  assimilate  recommendations 

6 

4 

3.56 

Previous  experience  with  simulations 

7 

4 

3.67 

Superior’s  acceptance  of  simulation  results 

8 

3 

3.11 

Verification  of  correct  software  coding 

9 

2 

2 . 33 

Degree  (magnitude)  of  effort  developing  results 

10 

2 

2.22 

Cost  of  getting  results 

11 

3 

2.67 

Other  program  managers'  acceptance  of  simulation 
results 

12 

2 

2.56 

Formal  education  courses  in  computer  simulation 

13 

3 

3.33 

Tact  used  in  present ing  results 

14 

4 

4.11 

Managers*  insights  and  intuition 
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that  counter  our  intuition  are  often  rejected.”  The  remaining  six 
factors  were  rated  as  approximately  equivalent  in  importance.  It 
appears  that  the  factors  ranked  numbers  one  through  five  have  equally 
strong  effect  on  the  managers  acceptance  of  simulation  results. 

Of  the  fourteen  factors  in  the  first  round  survey,  six  were  related 
to  the  manager's  own  perceptions.  The  eight  remaining  factors  pertain 
to  knowledge  required  by  managers  to  make  decisions.  These  eight  fac¬ 
tors  are  listed  in  Table  3.  In  round  one,  the  experts  were  asked  to 
mark  if  they  believed  managers  required  personal  knowledge  of  any  of 
these  factors.  Each  of  the  top  five  factors  in  Table  3  received  a  50 
percent  or  better  response.  These  top  five  factors  were  also  rated  as 
having  a  "strong  effect"  by  the  Likert  evaluation.  The  remaining  three 
factors  received  less  than  a  50  percent  response  when  the  experts  were 
asked  about  personal  knowledge  requirements  and  had  less  than  a  "strong 
effect"  Likert  median.  These  results  show-  that  experts  believe  that 
managers  require  personal  knowledge  of  those  factors  that  strongly 
affect  their  decisions.  Additionally,  the  experts  believe  that  the 
manager  expects  his  subordinates  to  monitor  the  less  important  factors 
as  indicated  by  the  lower  Likert  median  and  ranking.  In  other  words 
words,  managers  need  to  have  more  personal  knowledge  about  the  important 
factors  than  they  do  about  the  factors  which  have  little  effect,  so 
that  the  manager  can  focus  on  the  more  important  issues. 

A  number  of  very  good  comments  were  received  as  part  of  the  first 
round  responses  (Appendix  C) .  Several  of  these  comments  are  provided 
here.  One  expert  made  a  statement  in  connection  with  a  manager’s  under¬ 
standing  of  the  technical  information  used  in  simulations  that  "too 
often  managers  believe  the  data  is  being  manipulated  to  get  an  answer  if 


24 


Table  3.  Ranked  Factors  Requiring  Manager’s  Personal  Knowledge 


Persona  1 
Know 1 edge 


Rank 

Required 

Factor 

1 

89% 

Degree  to  which  model  represents  the  "real  world" 

2 

100% 

Technical  information  incorporated  in  the  computer 
model 

7 

50% 

Superior’s  acceptance  of  simulation  results 

8 

56% 

Verification  of  correct  software  coding 

9 

56% 

Degree  (magnitude)  of  effort  developing  results 

10 

11% 

Cost  of  getting  results 

11 

33% 

Other  program  managers’  acceptance  of  simulation 
results 

13 

0% 

Tact  used  in  presenting  results 

they  don’t  understand  the  detail.”  Another  expert  pointed  out  that  a 
model’s  ability  to  represent  the  "real  world"  is  not  always  very  impor¬ 
tant.  He  said  "many  times  we  can  only  guess  at  ’real  world’  but  the 
simulation  is  still  valuable  in  choosing  between  alternatives...."  Two 
of  the  more  significant  comments  were  recommendations  of  additional 
factors  to  be  included  in  the  second  round  that  had  not  been  included  in 
Wi 1 1 iam  Morris’  list,  described  in  Chapter  II.  The  first  recommended 
factor  was  "acceptance  by  user  [the  organization  for  which  the  weapon 
system  is  being  procured]  of  the  simulation  representing  the  real 
world.”  The  second  additional  factor  recommended  was  "Acceptability  of 
the  model  by  the  community  that  generally  conducts  simulations  in  the 
specialty  area."  These  factors  were  added  to  the  round  two  survey. 

Round  Two  The  round  two  survey  was  sent  to  each  of  the  nine  panel 
members  remaining  after  the  first  round.  Eight  of  the  nine  returned  the 
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survey  resulting'  in  an  89  percent  response  rate.  The  one  individual 
that  failed  to  return  the  survey  was  contacted.  He  stated  that  he  had 
not  understood  that  the  package  was  the  second  round  survey  and  had 
thrown  it  away  thinking  that  it  was  another  copy  of  round  one.  He  also 
said  that  he  did  not  have  the  time  to  complete  another  survey  and  asked 
to  be  removed  from  the  exercise. 

As  part  of  the  second  round  survey,  each  panel  member  was  asked  to 
respond  to  the  same  fourteen  Likert  scale  questions  plus  the  one  addi¬ 
tional  question  for  each  of  the  two  new  factors  introduced  as  a  results 
of  the  first  round  comments.  Each  of  the  original  fourteen  questions 
included  a  statement  of  his/her  first  round  response,  a  frequency  dis¬ 
tribution  of  all  first  round  responses  and  any  comments  which  had  been 
made.  This  information  allowed  the  individual  to  compare  his/her 
responses  to  those  of  the  other  members  of  the  panel.  A  summary  of  the 
second  round  Likert  questions  including  calculated  mean,  median  and 
statement  of  consensus  for  each  factor  are  shown  in  Table  4.  The  data 
derived  from  the  rank  ordering  question  was  includes  Ir  the  second  round 
survey.  The  mean  of  the  first  round  responses  for  each  factor  was  used 
as  the  biisis  to  rank  and  reorder  the  factors.  This  reordered  list  of 
factors  was  presented  to  the  panel  members  for  re-evaluation;  however, 
their  first  round  responses  were  not  included.  The  panel  members  were 
also  asked  to  place  the  two  new  factors  into  the  revised  ranking.  The 
revised  ranking  is  shown  in  Table  4  and  includes  the  calculated  mean  and 
median  from  the  Likert,  questions,  Table  5,  for  each  factor.  Comments 
returned  with  the  second  round  are  listed  in  Appendix  C,  along  with  the 
first  round  c  ommen  t s . 
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Table  4.  Likert  Responses  --  Round  Two  Delphi  Survey 


Rat ings 


Factor 

5 

4 

3 

2 

1 

9 

Mean 

Median 

Under standing  of 
technical  information 

1 

5 

2 

9 

9 

9 

3.81 

4 

Consensus 

Degree  represents 
"real  world" 

2 

6 

9 

9 

1 

9 

4.25 

4 

Consensus 

Code  verification 

9 

2 

3 

3 

9 

9 

2.88 

3 

Agreement  with  personal 
ins ight s/ intuit  ion 

3 

4 

1 

9 

9 

9 

4.25 

4 

Consensus 

Supervisor  acceptance 

1 

4 

3 

9 

9 

9 

3 .75 

4 

Consensus 

Other  program  managers 
acceptance 

9 

1 

4 

3 

9 

9 

2.75 

3 

Consensus 

Degree  (magnitude)  of 
development  effort 

9 

1 

3 

3 

1 

9 

2.59 

2.5 

Ability  to  understand 
recommendations  from 
results 

1 

7  . 

9 

9 

9 

1 

4.25 

4 

Consensus 

Confidence  in  individual 
presenting  results 

1 

7 

9 

9 

9 

9 

4.13 

4 

Consensus 

Confidence  in  individual 
performing  simulation 

1 

6 

9 

1 

9 

9 

3.88 

4 

Consensus 

Cost  of  performing 
s  irnulat  ion 

9 

9 

3 

4 

1 

9 

2.25 

2 

Consensus 

Personal  previous 
experience 

9 

5 

3 

9 

9 

9 

3.63 

4 

Consensus 

Skill  used  in 
presenting  results 

9 

3 

5 

9 

9 

9 

3.38 

3 

Consensus 

Formal  education 

courses 

9 

9 

3 

5 

9 

9 

2.38 

2 

Consensus 

Acceptance  by  user 

9 

6 

2 

9 

9 

9 

3.75 

4 

Consensus 

Acceptance  by 
simulation  community 

9 

7 

1 

9 

9 

9 

3.88 

4 

Consensus 
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Table  5.  Ranked  Factors  --  Round  Two  Delphi  Survey 


Likert 


Rank 

Med i an 

Mean 

Fac  t  or 

1 

4 

4.25 

Degree  to  which  model  represents  the  "real  world" 

2 

4 

3.81 

Technical  information  incorporated  in  the  computer 
model 

3 

4 

3.75 

Acceptance  by  user  of  the  simulation’s  ability  to 
represent  the  "real  world" 

4 

4 

3.88 

Confidence  in  individual  performing  simulation 

5 

4 

4.25 

Ability  to  assimilate  recommendations 

6 

4 

3.88 

Acceptability  of  model  by  the  community  that 
generally  conducts  simulations  in  the  specialty  area 

7 

4 

4.13 

Confidence  in  individual  presenting  results 

8 

4 

3.63 

Previous  experience  with  simulations 

9 

4 

3.75 

Superior’s  acceptance  of  simulation  results 

10 

4 

4.25 

Managers’  insights  arid  intuition 

11 

3 

2.88 

Verification  of  correct  software  coding 

12 

2 

2.25 

Cost  of  getting  results 

13 

2.5 

2 . 50 

Degree  (magnitude  >  of  effort  developing  results 

14 

3 

2.75 

Other  program  managers’  acceptance  of  simulation 
resul ts 

15 

2 

2.38 

Formal  education  courses  in  computer  simulation 

16 

3 

3.38 

Tact  used  in  presenting  results 
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The  second  round  of  the  Delphi  exercise  ended  with  consensus  being 
achieved  in  fourteen  of  the  sixteen  Likert  questions.  This  87  percent 
consensus  exceeds  the  7Q  percent  put  forth  in  the  previous  chapter  to 
determine  closure,  therefore  the  Delphi  exercise  was  not  carried  into  a 
third  round. 

A  comparison  of  the  medians  for  each  factor,  included  in  both  survey 
rounds,  shows  that  none  of  the  values  changed.  This  combined  with  the 
minor  changes  seen  in  the  calculated  Likert  means  indicate  that  no 
effective  change  in  opinion  took  place  during  the  two  rounds.  The 
ranking  question,  however,  included  a  significant  change  in  the  "man¬ 
ager’s  insights  and  intuition"  factor.  This  factor  was  moved  up  in  the 
ranking  six  positions.  This  move  indicates  a  definite  change  in  the 
expert  opinion  between  the  two  rounds.  The  move  resulted  in  the  factor 
being  positioned  more  appropriately  in  comparison  to  the  other  factors 
and  their  medians.  Most  of  the  other  factors  were  only  changed  by  one 
or  two  positions,  excluding  the  two  new  factors.  These  changes  indicate 
minor  adjustments,  refinements,  in  the  ranking  and  no  substantive  change 
in  expert  opinion  is  expressed.  The  second  round  ranking  resulted  in 
all  factors  with  median  of  "strong  effect"  being  in  the  top  ten  ranked 
posit  ions.  This  shows  that,  by  the  two  evaluation  techniques,  Likert 
scale  and  ranking  process,  these  factors  have  the  most  effect  on  the 
acceptance  of  computer  simulation  results.  However,  comparison  of  the 
top  ten  factors’  mean  and  median  values  with  the  rankings  suggest  that 
the  mean  is  not  a  good  discriminator  of  importance  within  the  group. 

The  top  three  factors  involve  knowing  the  actual  c’naracterist  ics  of  the 
simulation,  not  the  type  of  factor  requiring  the  use  of  intuition  or 
personal  judgement.  "Facts”  are  most  important  in  accepting  results. 
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The  lover  ranked  factors,  "11"  through  "16",  appear  to  nave  consid¬ 
erably  less  effect  on  the  acceptance  of  simulation  results.  \  compari¬ 
son  of  the  calculated  mean  and  median  for  the  Likert  questions,  arid  the 
rankings  supports  this  assertion. 

The  tvo  new  factors  that  were  added  as  a  result  of  the  first  round's 
comments  had  a  definite  impact  on  the  rankings.  The  factor  dealing  with 
"user  acceptance"  vas  ranked  number  three  and  the  factor  addressing  the 
acceptability  of  the  model  by  the  community  that  generally  conducts 
simulations  in  the  specialty  area  vas  ranked  sixth  out  of  the  sixteen 
factors.  This  is  a  significant  change  from  Morris’  findings  about  the 
factors  affecting'  the  acceptance  of  industrial  engineering  recommend¬ 
ations.  The  difference  seen  in  "user  acceptance"  is  easily  explained; 
Morris'  research  vas  centered  on  civilian  industry  vhere  the  company 
itself  is  the  "user."  There  vas  no  need  to  convince  an  outside  agency 
of  the  model’s  accuracy.  The  second  nev  factor  can  be  described  in  much 
the  same  vay  as  the  first,  hovever  there  is  the  added  fact  that  in 
industrial  companies  most  of  the  modeling  and  simulation  effort  is 
conducted  by  a  central  modeling  office.  In  the  situation  vhere  one 
central  office  conducts  the  simulations,  there  is  no  other  "community" 
that  conducts  the  simulations.  The  importance  of  these  tvo  factors  on 
the  Air  Force  manager’s  acceptance  of  computer  simulation  results  shov 
that  the  concerns  and  needs  of  the  Air  Force  program  manager  are  differ¬ 
ent  than  that  of  the  manager  in  a  civilian  company.  The  civilian  man¬ 
ager  and  the  Air  Force  program  manager  do  not  make  decisions  using  the 
same  types  of  information.  This  statement  is  reinforced  by  the  low 
importance  of  cost  in  the  dec  is  inn  process. 
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Summary 

This  chapter  reported  the  results  of  the  research.  The  data 
presented  was  the  result  of  a  two  round  Delphi  exercise.  The  exercise 
was  terminated  at  the  end  of  the  second  round.  TTie  first  round  survey 
was  sent  to  twenty  Air  Force  colonels  with  sixteen  responding,  an  80 
percent  response  ra.te.  From  the  sixteen  responses,  nine  were  mialified 
to  participate  in  the  second  round  of  the  survey.  Of  the  nine  sent  the 
second  ro  .'d  survey,  eight  responded  yielding  an  89  percent  response 
rate.  The  Delphi  exercise  was  terminated  at  the  end  of  the  second  round 
with  07  percent  of  the  questions  achieving  consensus.  The  first  round 
of  the  survey  resulted  in  the  addition  of  two  new  factors  to  the  second 
round.  The  second  round  ended  with  the  top  ten  ranked  factors,  in  Table 
5,  all  reflecting  a  "strong  effect"  on  the  acceptance  of  computer  simu¬ 
lation  results.  The  remaining  six  factors  show  a  smaller  effect  on 
simulation  result  acceptance.  The  final  chapter  presents  the  answers  to 
the  research  questions  and  the  effect  of  these  answers. 
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IV.  Cone lus ions  and  Reeommendat ions 

The  DC®  has  stated,  in  a  plan  given  to  Congress,  that  simulation  and 
modeling  are  essential  to  the  "long  term  qualitative  superiority  of  US 
weapon  systems"  (15:25).  For  these  simulations  and  models  to  be  useful 
two  things  must  be  true.  First,  the  simulations  and  models  must  be 
accurate.  If  the  simulations  and  models  are  not  accurate,  the  erroneous 
results  obtained  will  mislead  decision  makers  and  planners.  Secondly, 
these  simulations  and  models  must  be  accepted  by  management.  If  the 
simulations  and  models  are  perfectly  accurate,  but  the  managers  do  not 
accept  the  results,  they  are  not  used  by  the  organization.  This 
research  has  focused  on  managerial  acceptance  of  computer  simulation 
results  by  senior  program  managers  within  AFSC. 

In  the  area  of  simulation  accuracy,  there  is  a  multitude  of  litera¬ 
ture  and  research  being  performed.  However,  in  the  area  of  managers 
acceptance  of  simulation  results,  there  is  a  noticeable  lack  of  informa¬ 
tion  currently  available.  This  is  even  true  in  the  most  basic  part  of 
research,  the  determination  of  the  factors  affecting  acceptance  of 
computer  simulation  results.  Robert  Sargent  has  written  on  the  topic  of 
creating  accurate  simulations.  He  pointed  out  that  the  primary  factor 
determining  an  individual’s  confidence  in  simulation  is  the  degree  of 
validation  that,  has  been  performed  (20:33,38).  This  can  be  applied  to 
the  manage:'’ s  acceptance  of  simulation  results  as  well.  In  addition  to 
Sargent’s  writings,  William  Morris  has  written  on  managerial  acceptance 
of  industrial  engineering  recommendations.  Morris  agrees  with  Sargent 
♦.hat  validity  is  a  prime  factor  in  managerial  acceptance  of  information. 
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However,  Morris  also  proceeded  to  outline  eleven  additional  factors  that 
need  to  be  considered  (14:16-19). 

In  light  of  the  fact  that  no  prior  research  with  regard  to  Air  Force 
program  managers  was  found,  this  research  is  exploratory  in  nature.  The 
Delphi  survey  technique  of  data  collection  was  selected  to  identify  the 
factors  affecting  acceptance  of  computer  simulation  results.  The  fac¬ 
tors  outlined  by  Sargent  and  Morris  were  used  as  the  basis  for  this 
e:<p ! oral orv  research. 

The  first  round  Delphi  survey  was  sent  to  twenty  Air  Force  colonels 
in  job  positions  involving  the  acquisition  of  weapon  systems.  Sixteen 
of  the  colonels  responded,  however  only  nine  were  qualified  to  be  a 
member  of  the  expert  panel.  Only  eight  of  the  nine  experts  responded  in 
the  second  round  Delphi  survey.  A  presentation  of  the  survey  results 
and  their  detailed  analysis  is  presented  in  Chapter  III  of  this  docu¬ 
ment  . 

Limi tat  ions 

The  accuracy  of  the  research  presented  is  limited  for  several 
reasons.  First,  this  research  is  the  initial  attempt  to  characterize 
the  factors  affecting  the  acceptance  of  simulation  results,  therefore 
duplication  of  these  research  results  are  not  available.  Through  dupli¬ 
cation  of  research,  higher  confidence  is  associated  with  the  research 
results.  Secondly,  the  number  of  the  Delphi  panel  members  was  less  than 
ten.  In  Delphi  exercises  where  the  panel  has  less  than  ten  members,  the 
results  are  considered  to  tie  less  reliable  than  those  with  ten  or  more 
because  of  the  restriction  on  differing  opinions  and  experiences  from 
which  to  draw.  The  next  reason  for  limited  accuracy  in  the  research 
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results  is  the  sampling1  technique  used  for  this  research.  All  of  the 
experts  were  dravn  from  AFSC/ASD;  no  expert  outside  AFSC/ASD  was  con¬ 
sidered  for  participation  in  the  panel.  The  inclusion  of  experts  out¬ 
side  of  AFSC/ASD  could  have  increased  the  diversity  of  background  of  the 
panel  members  as  well  as  increased  the  number  of  experts  participating. 

Discuss  ion 

The  information  resulting  from  the  Delphi  exercise  was  sufficient  to 
answer  the  two  research  questions  presented  in  Chapter  I.  Each  question 
is  addressed  in  the  following  paragraphs. 

The  first  research  question  was:  What  are  the  factors  that  affect 
the  acceptance  of  computer  simulation  results  used  by  senior  program 
managers  within  AFSC/ASD?  The  answer  to  this  question  was  determined 
through  the  use  of  the  iterative  feedback  feature  of  the  Delphi  survey. 
Initially,  Sargent’s  and  Morris’  factors  were  used  to  formulate  the 
survey  questions.  The  experts  were  then  asked  to  suggest  additional 
factors  that  affect  acceptance.  Two  additional  factors  were  identified 
through  this  process  yielding  a  total  of  sixteen  factors.  The  list  of 
factors  is  shown  in  Table  6. 

The  second  research  question  was:  What  is  the  relative  importance  of 
the  factors  identified  to  the  senior  program  managers  within  AFSC/ASD? 
This  question  was  answered  through  two  techniques.  The  first  technique 
involved  evaluating  the  individual  factors  against  a  Likert  scale.  By 
taking  the  median  of  the  Likert  responses  and  comparing  each  against  the 
other  factors’  median  a  relative  importance  cam  be  determined.  The 
second  technique  involved  a  rank  ordering  of  the  factors  by  each  expert. 
This  technique  yielded  a  direct  indication  of  relative  importance.  The 
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Table  6.  Final  Ranked  Factors 


Likert 
Rank  Med  i  an 

1  4 

2  4 

3  4 

4  4 

5  4 

6  4 

7  4 

8  4 

9  4 

IQ  4 

11  3 

12  2 

13  2.5 

14  3 

15  2 

16  3 


Factor 

Degree  to  vhich  model  represents  the  "real  world" 

Technical  information  incorporated  in  the  computer  model 

Acceptance  by  user  of  the  simulation’s  ability  to 
represent  the  "real  world" 

Confidence  in  individual  performing  simulation 
Ability  to  assimilate  recommendations 

Acceptability  of  model  by  the  community  that  generally 
conducts  simulations  in  the  specialty  area 

Confidence  in  individual  presenting  results 

Previous  experience  with  simulations 

Superior’s  acceptance  of  simulation  results 

Managers*  insights  and  intuition 

Verification  of  correct  software  coding 

Cost  of  getting  results 

Degree  (magnitude)  of  effort  developing  results 
Other  program  managers’  acceptance  of  simulation  results 
Formal  education  courses  in  computer  simulation 
Tact  used  in  presenting  results 
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result  of  these  techniques  are  shown  in  Table  6.  The  results  show  that 
the  top  ten  ranked  factors  also  have  the  highest  medians.  The  fact  that 
both  techniques  are  in  agreement  gives  strong  indication  that  the  fac¬ 
tors  are  correctly  grouped  together  and  that  they  are  the  primary 
influencing  forces  in  the  acceptance  of  simulation  results.  The  remain¬ 
ing  six  factors  also  show  good  agreement  between  the  Likert  and  ranking 
techniques.  The  ranking  and  medians  indicate  that  these  factors  have  a 
much  lower  influence  on  the  acceptance  of  simulation  results  and,  there¬ 
fore,  should  not  be  regarded  as  heavily  as  the  top  ranked  factors. 

According  to  the  research,  for  computer  simulation  results  to  be 
accepted  by  top  management,  the  simulations  must  correctly  represent  the 
"real  world"  and  the  manager  should  agree  with  the  technical  information 
incorporated  into  the  simulation.  Additionally,  he/she  should  believe 
the  final  user  of  the  weapon  system  would  accept  the  simulation  as  a 
good  representation  of  the  "real  world"  and  the  portion  of  the  modeling 
community  that  generally  conducts  these  simulations  would  also  accept 
the  models  used.  The  manager  himself  must  have  confidence  in  the  indi¬ 
vidual  performing  the  simulation,  be  able  to  assimilate  the  recommenda¬ 
tions  resulting  from  the  simulations  and  have  confidence  in  the  individ¬ 
ual  presenting  the  results.  The  manager  also  takes  into  consideration 
his/her  prior  experience  with  simulations,  his/her  superior’s  acceptance 
of  simulation  results,  and  his/her  own  personal  insights  and  intuition. 

It  is  obvious  from  this  research  that  managers,  according  to  the 
experts,  will  be  more  likely  to  use  the  results  of  computer  simulations 
if  those  factors  identified  above  are  considered  during  model  develop¬ 
ment.  Model  developers  might  want  to  consider  a  combination  of  actions 
to  improve  the  use  of  simulation  results.  First,  an  evaluation  of  the 
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simulation’s  technical  composition  and  accuracy  should  be  performed.  As 
part  of  this  evaluation,  the  community  that  generally  conducts  the 
specific  type  of  simulation  should  be  consulted  as  to  their  views  con¬ 
cerning  the  computer  model  under  consideration  for  use.  The  result  of 
this  evaluation  should  then  be  presented  to  the  decision  makers 
involved.  Secondly,  the  weapon  system  user,  the  SPO’s  customer,  should 
be  included  in  all  phases  of  the  computer  modeling  and  simulation  proc¬ 
ess.  By  involving  the  user  early  in  the  simulation  effort,  the  user’s 
needs  and  concerns  can  be  taken  into  account;  thereby  increasing  the 
overall  acceptance  of  the  simulation  results.  Next,  select  the  individ¬ 
ual  (s)  to  perform  the  simulation  on  the  basis  of  expertise  in  simula¬ 
tions  and  the  specialty  area  to  be  simulated.  Finally,  individual ( s ) 
presenting  information  concerning  the  simulation  or  its  results  should 
have  a  thorough  understanding  of  the  facts  surrounding  the  simulation 
effort  and  address  only  those  results  that  have  a  direct  bearing  on  the 
decision  at  hand. 

Recommendat ions 

The  following  recommendations  are  made  for  future  research  to  better 
define  the  status  of  computer  simulation  utilization  within  Air  Force 
weapon  system  procurement. 

1.  Research  should  be  undertaken  to  replicate  the  results  of  this 
study.  A  second  Delphi  exercise  should  be  performed  using  a  larger 
number  and  a  broader  set  of  experts.  The  research  should  increase 
confidence  in  the  factors  arrived  at  in  this  thesis  and  identify  other 
factors  that  may  not  have  been  introduced  by  this  panel  of  experts. 
Follow-on  Delphi  exercises  should  probe  experts  from  other  AFSC  product 
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divisions,  other  major  commands  and  higher  headquarters.  By  probing 
these  varied  groups,  a  generalized  model  of  factors  affecting  acceptance 
of  computer  simulation  results  can  be  developed. 

2.  Research  should  be  undertaken  to  determine  the  current  level  of 
acceptance  of  computer  simulation  results  by  managers  Air  i-orce  wide. 

The  sample  should  include  individuals  from  various  major  commands,  as 
well  as  acquisition  and  test  organizations.  The  research  should  show 
any  issues  or  problems  that  may  exist  and  indicate  where  training  and/or 
education  is  needed  to  improve  and  increase  the  usage  of  computer  simu¬ 
lations  by  managers. 

3.  This  thesis  indicated  that  44  percent  of  the  managers  surveyed 
had  never  used  simulation  results  in  decision  making.  Research  should 
be  undertaken  to  identify  program  managers  who  state  they  have  never 
used  computer  simulation  results  in  decision  making.  The  research  will 
include  a  determination  as  to  the  correctness  of  the  statement  as  well 
as  a  determination  of  why  computer  simulation  has  not  been  used.  This 
research  should  uncover  areas  where  program  managers  are  unknowingly 
using  simulation  results  and  those  where  the  manager  is  intentionally 
avoiding  the  use  of  computer  simulation  results.  In  both  instances, 
computer  simulation  results  are  not  being  used  in  the  correct  or  most 
effective  manner. 

Summary 

The  thrust  of  this  research  was  to  determine  the  factors  which 
contribute  to  the  acceptance  of  computer  simulation  results.  This 
determination  was  accomplished  using  the  Delphi  data  collection  tech- 
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weapon  systems  for  the  L'SAF. 
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DEPARTMENT  OF  THE  AIR  FORCE 

AIR  UNIVERSITY 

AIR  FORCE  INSTITUTE  OF  TECHNOLOGY 
WRIGHT-PATTERSON  AIR  FORCE  BASE  OH  45433-6533 

Nppendix  A:  Round  (jie  Delphi  Survey 

11  May  1989 


&NAME& 

CxOFFlCtc* 

Wright- Patterson  AFB,  OH  45433-6583 
Dear  &NAMD& : 

AFIT  is  performing  a  Delphi  survey.  The  purpose  of  this  research  is 
to  determine  the  factors  which  affect  managers'  acceptance  of  computer 
simulation  results.  You  have  been  selected  to  participate  in  this 
important  research  because  your  experience  and  insight  qualify  you  as  an 
"expert"  in  program  management.  Your  opinions  ana  comments  will  be 
combined  with  those  of  other  "experts"  to  develop  a  descriptive  model  of 
factors  affecting  acceptance  of  simulation  results. 

The  attached  Delphi  survey  solicits  your  personal  opinions  in  a 
number  of  areas.  To  assist  in  this  research,  please  complete  the  survey 
and  return  it  within  1.0  days.  As  soon  as  all  the  responses  are 
compiled,  a  second  survey  will  be  sent  to  you. 

Your  comments,  suggestions,  and  ideas  regarding  this  research  and 
the  model  are  welcome  and  encouraged.  If  you  have  any  questions  about 
this  survey  please  call  me  at  (513)  255-3355  (AV : 785- 3355)  or  Captain 
Thomas  Wiggs  at  (513)  255-5435  (AV: 785-5435) .  Thank  you  for  making  time 
to  share  your  expertise. 


2  Atchs 
Delphi  Survey 
Envelope 
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JOHN  DUMOND,  Lt  Col,  USAF 
Head,  Department  of  System 
Acquisition  Management 
School  of  Systems  and  Logistics 


Round  One  Delphi  Survey 

1.  Survey  Objective: 

To  obtain  expert  opinion  on  what  factors  affect  the  managers’ 
acceptance  of  computer  simulation  results. 

2 .  Gene  r a  1  Commen  t  s : 

a.  The  subject  areas  covered  in  this  questionnaire  are  not  meant 
to  be  complc+e  or  exhaustive.  Instead,  the  coverage  is  designed  to 
stimulate  your  thinking. 

b.  Your  participation  and  honest  opinions  are  key  to  the  success 
of  this  research  project.  There  are  no  right  or  wrong  answers . 
Therefore,  all  your  ideas  and  brainstorming  comments  should  be  included. 
In  later  rounds  of  questioning,  these  ideas  may  spark  additional 
comments  by  other  part ic ipants . 

c.  At  least  two  rounds  of  questions  will  be  needed  to  arrive  at  a 
group  consensus.  The  first  round  should  not  take  more  than  twenty 
minutes  to  complete.  Each  of  the  remaining  one  or  two  rounds  should 
take  less  than  thirty  minutes.  After  each  round,  all  participants’ 
responses  will  be  compiled  and  given  back  to  you  at  the  „t«rt  of  the 
next  round.  You  will  be  provided  an  executive  summary  of  this  research 
after  it  is  completed. 

d.  The  questionnaire  is  divided  into  two  sections.  The  first  set 
of  questions  are  inquires  as  to  your  personal  background.  The  second 
set  of  questions  address  the  issue  of  factors  af fee* >ng  managers ' 
acceptance  of  computer  simulation  results. 

e.  The  number  in  the  upper  right-hand  comer  of  the  questionnaire 
is  for  survey  control  purposes  only.  Please  be  assured  that  complete 
anonym i ty  wi 1 1  be  enforced. 

2.  Specific  Instructions: 

a.  When  the  question  calls  for  an  answer  along  a  scale,  please 
circle  the  response  which  most  accurately  reflects  your  judgment  on 
that  (Question. 

b.  When  a  question  requires  a  ranking  response,  please  rank  order 
the  alternatives,  using  "1”  for  the  most  important  item. 

c.  Please  feel  free  to  include  the  rationale  for  your  answers, 
especially  for  those  areas  where  you  feel  strongTy.  Add  any 
illustrations,  examples,  or  experiences  you  have  had  that  will  help  the 
other  participants  understand  your  response.  Feel  free  to  write  your 
comments  on  the  back  of  the  survey  sheets.  Please  number  your  comments 
so  they  correspond  to  the  question  you  are  answering. 
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d.  Any  ideas  or  recommendations  you  have  for  improving  the 
manager’s  acceptance  of  computer  simulation  results  should  also  be 
included  with  your  responses.  Your  ideas  will  be  shared  with  others  who 
care  about  the  acceptance  of  simulation  results. 

e.  The  last  page  of  this  survey  is  for  any  additional  conments  you 
feel  are  pertinent  to  this  study. 

f.  Tf  vou  have  any  questions  about  this  survey  please  call  Capt 
Thomas  Wiggs  at  (513)  255-5435  (AV:  785-5435)  or  LtCol  John  Dumond  at 
(513)  255-3355  (AV:  785-3355). 


THANK  YOU  FOR  PARTICIPATING  IN  THIS  SURVEY. 


Sec  t i on  1 : 


Questions  on  your  persona]  background. 


What  is  your  RANK/GRADE:  ______ 

What  was  your  major  field  of  study  for  each  degree: 

(Place  answer  in  the  space  provided  next  to  each  degree  listed 
be 1 ov ) 

A.  Engineering 

B.  Liberal  .Arts 

C.  Sciences 

D.  Business  Administration 

E.  Program  Management 

F.  Other: 


Bachelor:  _ 

Masters:  _ 

Doct  orate : 

3.  How  many  formal  computer  simulation  courses  have  you  taken: 


4.  How-  many  years  of  System  Program  Office  (SPO)  experience  do  you 
have?  (To  the  nearest  year): 


5.  How  many  years  of  computer  simulation  experience  do  you  have?  Count 
the  years  of  development,  design  and/or  programming  of  simulations, 
as  veil  as  decision  making  based  on  simulations.  (To  the  nearest 
year )  : 


6.  What  major  commands  have  you  been  assigned  to: 

(Mark  all  that  apply) 

A.  SAC  C.  MAC  E.  AFLC  G.  Other  _ 

B.  TAC  D.  AFSC  F.  AFCC 

7.  Have  you  ever  participated  in  the  development  a  computer  model? 

_  YES  _  NO 

8 .  Have  you  ever  used  the  results  of  a  computer  simulation  in  decision 
making? 

_  YES  _  NO 

9.  If  the  answer  to  both  questions  7  and  8  above  are  NO,  DO  NOT  GO  ANY 
FURTHER.  Please  fold  the  ques t i ona i re ,  place  it  in  the  provided 
envelope  and  return  to  the  address  provided. 
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Section  2:  Prior  research  has  suggested  that  factors  in  the  questions 

below  may  affect  the  acceptance  of  computer  simulation  results  by 
managers.  Please  read  each  question  and  circle  the  answer  that  most 
accurately  reflects  your  judgment  on  that  question.  Use  the  following 
scale : 

VERY'  STRONG  STRONG  MODERATE  LITTLE  NO  NO  OPINION/ 

EFFECT  EFFECT  EFFECT  EFFECT  EFFECT  DON’T  UNDERSTAND 


IQ.  What  effect  does  a  manager’s  understanding  of  the  technical 
information  used  in  simulations  have  on  their  acceptance  of 
simulation  results? 

YERY  STRONG  STRONG  MODERATE  LITTLE  NO  NO  OPINION/ 

EFFECT  EFFECT  EFFECT  EFFECT  EFFECT  DON’T  UNDERSTAND 


11.  What  effect  does  the  degree  to  which  a  computer  model  represents  the 
"real  world"  have  on  a  manager's  acceptance  of  simulation  results" 

YERY  STRONG  STRONG  MODERATE  LITTLE  NO  NO  OPINION/ 

EFFECT  EFFECT  EFFECT  EFFECT  EFFECT  DON’T  UNDERSTAND 


12.  What  effect  does  the  verification  of  correct  software  coding  in  the 
model  used  for  simulation  have  on  a  manager’s  acceptance  of 
simulation  results? 

YERY  STRONG  STRONG  MODERATE  LITTLE  NO  NO  OPINION/ 

EFFECT  EFFECT  EFFECT  EFFECT  EFFECT  DON’T  UNDERSTAND 


13.  What  effect  does  the  simulation  results’  agreement  with  the 

manager’s  insights  and  intuition  have  on  his/her  acceptance  of 
these  results? 

YERY  STRONG  STRONG  MODERATE  LITTLE  NO  NO  OPINION/ 

EFFECT  EFFECT  EFFECT  EFFECT  EFFECT  DON’T  UNDERSTAND 


Comment  s : 
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1-4.  What  effect  does  the  general  acceptance  of  simulation  results  by  a 
manager’s  superiors  have  on  his/her  acceptance  of  simulation 
resul t  s? 


VERY  STRONG  STRONG  MODERATE  LITTLE  NO  NO  OPINION/ 

EFFECT  EFFECT  EFFECT  EFFECT  EFFECT  DON’T  UNDERSTAND 


15.  'what  effect  does  the  acceptance  of  simulation  results  by  other 
program  managers  have  on  a  manager’s  acceptance  of  simulation 
resul ts? 

\ERY  STRONG  STRONG  MODERATE  LITTLE  NO  NO  OPINION/ 

EFFECT  EFFECT  EFFECT  EFFECT  EFFECT  DON’T  UNDERSTAND 


16.  What  effect  does  the  degree  (magnitude)  of  effort  used  in  developing 
the  simulation  have  on  a  manag'er’s  acceptance  of  the  results? 

\ERY  STRONG  STRONG  MODERATE  LITTLE  NO  NO  OPINION/ 

EFFECT  EFEECT  EFFECT  EFFECT  EFFECT  DON’T  UNDERSTAND 


17.  What  effect  does  rhe  manager's  ability  to  unde rst. and  recommendations 
resulting  from  simulations  have  on  his/her  acceptance  of  the 
simulation  results? 

VERY  STRONG  STRONG  MODERATE  LITTLE  NO  NO  OPINION/ 

EFFECT  EFFECT  EFFECT  EFFECT  EFFECT  DON’T  UNDERSTAND 


18.  What  effect  does  a  manager’s  confidence  in  the  individual  presenting 
the  results  have  on  his/her  acceptance  of  simulation  results? 


VERY  STRONG 
EFFECT 


STRONG  MODERATE  LITTLE  NO 

EFFECT  EFFECT  EFFECT  EFFECT 


NO  OPINION/ 

DON’T  UNDERSTAND 


C'i  jinmen  t  s  : 
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19.  What  effect  does  confidence  in  the  individual  performing  the 

simulation  have  on  a  manager’s  acceptance  of  simulation  results? 

\ERY  STRONG  STRONG  MODERATE  LITTLE  NO  NO  OPINION/ 

EFFECT  EFFECT  EFFECT  EFFECT  EFFECT  DON’T  UNDERSTAND 


20.  What  effect  does  the  cost  of  obtaining  the  simulation  have  on  a 
manager’s  acceptance  of  simulation  results? 

VERY  STRONG  STRONG  MODERATE  LITTLE  NO  .NO  OPINION/ 

EFFECT  EFFECT  EFFECT  EFFECT  EFFECT  DON’T  UNDERSTAND 


21.  What  effect  does  a  manager's  previous  experience  with  simulation 
have  on  their  acceptance  of  simulation  results  later? 

VERY  STRONG  STRONG  MODERATE  LITTLE  NO  NO  OPINION/ 

EFFECT  EFFECT  EFFECT  EFFECT  EFFECT  DON’T  UNDERSTAND 


22.  What  effect  does  skill  used  in  presenting  the  simulation  results 
have  on  a  manager's  acceptance  of  these  results? 

VERY  STRONG  STRONG  MODERATE  LITTLE  NO  NO  OPINION/ 

EFFECT  EFFECT  EFFECT  EFFECT  EFFECT  DON’T  UNDERSTAND 


23.  What  effect  does  formal  education  courses  in  computer  simulation 
have  on  a  manager’s  acceptance  of  simulation  results? 


VERY  STRONG  STRONG  MODERATE 

EFFECT  EFFECT  EFFECT 


LITTLE  NO 
EFFECT  EFFECT 


NO  OPINION/ 

DON’T  UNDERSTAND 


Comments : 
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24.  In  the  blanks  to  the  left  of  each  item,  rank  order  the 

following  items  (1  being  most  important)  as  to  what  you  feel  has  had 
the  most  impact  on  managers’  acceptance  of  computer  simulation 
resu Its. 

In  the  blanks  to  the  right  of  each  item,  mark  the  items  that 
managers  require  personal  knowledge  of,  leaving  blank  those  items 
that  the  manager  does  not  care  about  or  that  may  be  the 
responsibility  of  individuals  lower  in  the  organizational  structure. 

PERSONAL 

RANK  KNOWLEDGE 

REQUIRED 

_ A.  Technical  information  incorporated  in  _ 

the  computer  model 

_ B.  Degree  to  which  model  represents  the  _ 

’’real  world" 

_ C.  Verification  of  correct  software  coding  _ 

_ D.  Managers’  insights  and  intuition  _ 

_ E.  Superior’s  acceptance  simulation  results  ___ 

_ _F.  Other  program  managers’  acceptance  of 

simulation  results 

G.  Degree  (magnitude)  of  effort  developing  _ 

resul t  s 

_ H.  Ability  to  assimilate  recommendations  _ 

_ I.  Confidence  in  individual  presenting  _ 

resu  1  i s 

_ J.  Confidence  in  individual  performing  _ 

s imulat ion 

_ K.  Cost  of  getting  results  _ 

_ L.  Previous  experience  with  simulations  _ 

_ M.  Tact  used  in  presenting  results  _ 

_ N.  Formal  education  courses  in  computer  _ 

s imulat ion 


Comment  s : 
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Is  there  anything  else  you  would  like  to  add  or  any  other  factors  you 
feel  should  be  included? 


Thank  you  for  completing  this  questionnaire  and  sharing  your  opinions. 
Please  send  through  distribution  or  mail  this  survey  as  soon  as  possible 
to: 

AFIT/LSG  (Bldg  641)  (Capt  Thomas  Wiggs) 

Wr ight -Patterson  AFB,  OH  45433-6583 
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DEPARTMENT  OF  THE  AIR  FORCE 

AIR  UNIVERSITY 

AIR  FORCE  INSTITUTE  OF  TECHNOLOGY 
WRIGHT-PATTERSON  AIR  FORCE  BASE  OH  45433-6583 

Appendix  B:  Round  Two  Delphi  Survey 


21  June  1989 


&NAME& 

&OFFICE& 

Wright -Patterson  AFB,  OH  45433-6583 
Dear  &NAME&: 

Thank  you  for  completing  the  first  round  of  the  AFIT  Delphi  survey 
on  the  factors  affecting  acceptance  of  computer  simulation  -3sults. 

Your  comments  were  of  great  value  to  this  research. 

The  second  round  Delphi  questionnaire  containing  respondent  feedback 
is  attached.  Please  read  the  comments  and  then  answer  the  questions 
that  follow.  You  will  also  note  that  the  feedback  provided  for  each 
question  includes  the  frequency  distribution  of  responses,  plus  your 
response  on  the  last  Delphi  questionnaire.  You  may  want  to  consider  all 
the  feedback  in  making  your  responses  on  this  questionnaire. 

Captain  Wiggs  and  I  appreciate  the  time  you  are  investing  in  this 
research.  Please  try  to  return  your  completed  survey  within  two  weeks, 
so  the  responses  can  be  analyzed  and  a  third  round  begun  in  July,  if 
needed.  Thank  you  again  for  helping  us  learn  more  about  the  usage  of 
computer  simulation  results. 


JOHN  DUMOND,  Lt  Col,  USAF 
Head,  Department  of  System 
Acc  iisition  Management 
School  of  Systems  and  Logistics 


2  Atchs 
Delphi  Survey 
Envelope 
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M  XNO-VLLOGE 


Round  Two  Delphi  Survey 


1.  Survey  Objective: 

To  obtain  expert  opinion  on  what  factors  affect  the  managers’ 
acceptance  of  computer  simulation  results. 

2 .  Genera 1  Comment  s : 

a.  During  this  round  you  wri  1 1  be  given  the  frequency  distribution 
for  all  responses  for  each  question  directly  above  the  descriptive 
terms.  You  will  also  have  representative  comments  listed  with  the 
question  for  which  it  was  given.  This  feedback  is  designed  to  provide 
some  "food  for  thought"  as  you  revisit  these  questions.  You  will  have 
space  to  make  further  comments  regarding  this  feedback. 

b.  Your  participation  and  honest  opinions  are  key  to  the  success 
of  this  research  project.  There  are  no  right  or  wrong  answers . 
Therefore,  all  your  ideas  and  brainstorming  comments  should  be  included. 

c.  The  number  in  the  upper  right-hand  comer  of  the  questionnaire 
; s  for  survey  control  purposes  only.  Please  be  assured  that  complete 
anonym i  t y  wi  1 1  be  enforced. 

3.  Specific  Instructions: 

a.  Please  consider  the  feedback  provided  with  each  question  before 
you  respond  to  the  question.  Please  note  that  as  a  result  of  round  one 
there  are  two  new  factors  which  require  evaluation. 

b.  When  the  question  calls  for  an  answer  along  a  s^sIp,  please 
circle  the  response  which  most  accurately  reflects  your  judgment  on 
that  question. 

c.  When  a  question  requires  a  ranking  response,  please  rank  order 
the  alternatives,  using  "1"  for  the  most  important  item.  On  question  17 
you  will  be  asked  to  show  where  the  two  new  factors  should  be  inserted 
into  the  order  of  importance. 

d.  Please  feel  free  to  include  the  rationale  for  your  answers, 
especially  for  those  areas  where  you  feel  strongly.  Add  any 
illustrations,  examples,  or  experiences  you  have  had  that  will  help  the 
other  participants  understand  your  response.  Feel  free  to  write  your 
comments  on  the  back  of  the  survey  sheets.  Please  number  your  comments 
so  they  correspond  to  the  question  you  are  answering. 

e.  Any  ideas  or  recommendations  you  have  for  improving  the 
manager’s  acceptance  of  computer  simulation  results  should  also  be 
included  with  your  responses.  Your  ideas  will  be  shared  with  others  who 
care  about  the  acceptance  of  simulation  results. 
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e.  The  last  page  of  this  survey  is  for  any  additional  conments  you 
feel  are  pertinent  to  this  study. 

f.  If  you  have  any  questions  about  this  survey  please  call  Capt 
Thomas  Wiggs  at  (513)  845-Q652  or  LtCol  John  Dumond  at  (513)  255-3355 
(AV:  785-3355). 


THANK  YOU  FOR  PARTICIPATING  IN  THIS  SURVEY. 


Prior  research  has  suggested  that  factors  in  the  questions  belov  may 
affect  the  acceptance  of  computer  simulation  results  by  managers. 

Please  read  each  question  and  circle  the  answer  that  most  accurately 
reflects  your  judgment  on  that  question.  Use  the  following 
scale:  , 

VERY  STRONG  STRONG  MODERATE  LITTLE  NO  '  NO  OPINION/ 

EFFECT  EFFECT  EFFECT  EFFECT  EFFECT  DON’T  UNDERSTAND 


1.  What  effect  does  a  manager’s  understanding  of  the  technical 
information  used  in  simulations  have  on  their  acceptance  of 
simulation  results? 

YOUR  ROUND  1  RESPONSE: 


FOMENTS:  "Between  moderate  and  strong  -  too  often  managers  believe 

the  data  is  being  manipulated  to  get  an  answer  if  they  don’t 
understand  the  detail."  (NOTE:  Data  point  counted  as  a  "Strong") 


ROUND  1  RESPONSES: 

1  6 

2 

9 

0 

9 

\ERY  STRONG  STRONG 

MODERATE 

LITTLE 

NO 

NO  OPINION/ 

EFFECT  EFFECT 

EFFECT 

EFFECT 

EFFECT 

DON’T  UNDERSTAND 

2.  What  effect  does  the  degree  to  which  a  computer  model  represents  the 
"real  world"  have  on  a  manager’s  acceptance  of  simulation  results? 

YOUR  ROUND  1  RESPONSE: 

COMMENTS:  "Depends:  many  times  we  can  only  guess  at  'real  world’ 

but  the  simulation  is  still  valuable  in  choosing  betv-een 
alternatives,  using  the  source  set  of  assumptions,  or  a  controlled 
set  of  variables." 

(NOTE:  One  panel  member  supplied  two  responses  in  connection  with 
the  above  comment ) 


ROUND  1  RESPONSES: 

4  5 

9 

9 

1 

9 

VERY  STRONG  STRONG 

MODERATE 

LITTLE 

NO 

NO  OPINION/ 

EFFECT  EFFECT 

EFFECT 

EFFECT 

EFFECT 

DON’T  UNDERSTAND 

Comments : 
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3.  What  effect  does  the  verification  of  correct  software  coding  in  the 
model  used  for  simulation  have  on  a  manager’s  acceptance  of 
simulation  results? 


YOUR  ROUND  1  RESPONSE: 


COMMENTS :  "Think  the 

answer  i s 

little, 

but  should  be 

!  strong." 

ROUND  1  RESPONSES: 

1  2 

3 

3 

© 

0 

\ERY  STRONG  STRONG 

MODERATE 

LITTLE 

NO 

NO  OPINION/ 

EFFECT  EFFECT 

EFFECT 

EFFECT 

EFFECT 

DON’T  UNDERSTAND 

4.  What  effect  does  the  simulation  results’  agreement  with  the 

manager’s  insights  and  intuition  have  on  his/her  acceptance  of 
these  results? 

YOUR  ROUND  1  RESPONSE: 

COMMENTS:  "Too  often  the  model  results  are  accepted  only  if  they 

support  intution." 

"Unfortunately,  we  all  suspect  the  model  builder,  because 
simulations  have  been  used  for  political  purposes;  thus,  and  again 
unfortunately,  simulations  that  counter  our  intution  are  often 
rejected. " 


ROUND  1  RESPONSES : 

3  : 

2 

0 

0 

0 

VERY  STRONG  STRONG 
EFFECT  EFFECT 

MODERATE 

EFFECT 

LITTLE 

EFFECT 

NO 

EFFECT 

NO  OPINION/ 

DON’T  UNDERSTAND 

Comments : 
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5.  What  effect  does  the  general  acceptance  of  simulation  results  by  a 
manager’s  superiors  have  on  his/her  acceptance  of  simulation 
results? 

YOUR  ROUND  1  RESPONSE: 

OOMENTS:  NONE 


ROUND  1  RESPONSES: 


1 

4 

4 

9 

9 

9 

VERY  STRONG 
EFFECT 

STRONG 

EFFECT 

MODERATE 

EFFECT 

LITTLE 

EFFECT 

NO 

EFFECT 

NO  OPINION/ 

DON’T  UNDERSTAND 

6. 


What  effect  does  the  acceptance  of  simulation  results  by  other 
program  managers  have  on  a  manager’s  acceptance  of  simulation 


YOUR  ROUND  1  RESPONSE: 


COMMENTS :  NONE 


ROUND  1  RESPONSES; 


9 

1 

4 

4 

9 

9 

VERY  STRONG 
EFFECT 

STRONG 

EFFECT 

MODERATE 

EFFECT 

LITTLE 

EFFECT 

NO 

EFFECT 

NO  OPINION/ 

DON’T  UNDERSTAND 

7-  What  effect  uOta  Liii-  degree  (magnitude)  of  effort  used  in  developing 
the  simulation  have  on  a  manager’s  acceptance  of  the  results? 

YOUR  ROUND  1  RESPONSE: 

OCMMENTS :  NONE 


ROUND  1  RESPONSES: 


9 

1 

3 

3 

2 

9 

\ERY  STRONG 
EFFECT 

STRONG 

EFFECT 

MODERATE 

EFFECT 

LITTLE 

EFFECT 

NO 

EFFECT 

NO  OPINION/ 

DON’T  UNDERSTAND 

Comments : 
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8.  What  effect  does  the  manager’s  ability  to  understand  recommendations 
resulting  from  simulations  have  on  his/her  acceptance  of  the 
simulation  results? 

YOUR  ROUND  1  RESPONSE: 


COMENTS:  NONE 


ROUND  1  RESPONSES: 

3  5 

0 

0 

0 

1 

VERY  STRONG  STRONG 

MODERATE 

LITTLE 

NO 

NO  OPINION/ 

EFFECT  EFFECT 

EFFECT 

EFFECT 

EFFECT 

DON’T  UNDERSTAND 

9.  What  effect  does  a  manager’s  confidence  in  the  individual  presenting 
the  results  have  on  his/her  acceptance  of  simulation  results? 

YOUR  ROUND  1  RESPONSE: 

COMENTS :  "The  simulator  better  be  able  to  defend  every  variable 

and  assumption.  Consistency  and  practicality  are  vhat  I  look  for. 
Did  he  change  his  model  for  this  simulation?  If  so  why?" 

ROUND  1  RESPONSES: 

2  6  1  0  9  9 

VERY  STRONG  STRONG  MODERATE  LITTLE  NO  NO  OPINION/ 

EFFECT  EFFECT  EFFECT  EFFECT  EFFECT  DON’T  UNDERSTAND 


10.  What  effect  does  confidence  in  the  individual  performing  the 

simulation  have  on  a  manager’s  acceptance  of  simulation  results? 

YOUR  ROUND  1  RESPONSE: 


OOMENTS:  "Cons  istency /Honesty" 


ROUND  1  RESPONSES: 

2  5 

1 

1 

0 

0 

VERY  STRONG  STRONG 
EFFECT  EFFECT 

MODERATE 

EFFECT 

LITTLE 

EFFECT 

NO 

EFFECT 

NO  OPINION/ 

DON’T  UNDERSTAND 

Comment  s : 
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11.  What  effect  does  the  cost  of  obtaining  the  simulation  have  on  a 
manager’s  acceptance  of  simulation  results? 

YOUR  ROUND  1  RESPONSE: 


OOMENTS:  NONE 


ROUND  1  RESPONSES: 


'L/  J.  ■  " 

9 

9 

4 

3 

2 

9 

VERY  STRONG 
EFFECT 

STRONG 

EFFECT 

MODERATE 

EFFECT 

LITTLE 

EFFECT 

NO 

EFFECT 

NO  OPINION/ 

DON’T  UNDERSTAND 

12.  What  effect  does  a  manager’s  previous  experience  with  simulation 
have  on  tneir  acceptance  of  simulation  results  later? 

YOUR  ROUND  1  RESPONSE: 


COMTEXTS : 

NONE 

ROUND  1  RESPONSES: 

9 

5 

4 

9 

9 

9 

VERY  STRONG 

STRONG 

MODERATE 

LITTLE 

NO 

NO  OPINION/ 

EFFECT 

EFFECT 

EFFECT 

EFFECT 

EFFECT 

DON’T  UNDERSTAND 

13.  What  effect  does  skill  used  in  presenting  the  simulation  results 
have  on  a  manager’s  acceptance  of  these  results? 

YOUR  ROUND  1  RESPONSE: 


COUMENTS :  NONE 


ROUND  1  RESPONSES: 
9  3 


6  9  9 


9 


VERY  STRONG  STRONG  MODERATE 

EFFECT  EFFECT  EFFECT 


LFTTLE  NO 
EFFECT  EFFECT 


NO  OPINION/ 

DON’T  UNDERSTAND 


Commen  t  s : 
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1-1.  What  effect  does  formal  education  courses  in  computer  simulation 
have  on  a  manager’s  acceptance  of  simulation  results? 

YOUR  ROUND  1  RESPONSE: 

OOMENTS :  NONE 


ROUT®  1  RESPONSES: 


9 

1 

3 

5 

9 

9 

\ERY  STRONG 

STRONG 

MODERATE 

LITTLE 

NO 

NO  OPINION/ 

EFFECT 

EFFECT 

EFFECT 

EFFECT 

EFFECT 

DON’T  UNDERSTAND 

15.  What  effect 

does  the 

acceptance 

by  t  he 

user  of  the 

s imulat ion’s 

ability  to  represent  the  "real  world"  have  on  a  manager’s  acceptance 
of  simulation  results? 

***  \pv  Quest  ion  *** 

VETO'  STRONG  STRONG  MODERATE  uiTTLE  NO  NO  OPINION/ 

EFFECT  EFFECT  EFFECT  EFFECT  EFFECT  DON’T  UNDERSTAND 


16.  What  effect  does  the  acceptability  of  the  model  by  the  community 
that  generally  conducts  simulations  in  the  specialty  area  (for 
instcince--  there  are  several  ECM  models,  but  none  are  widely- 
accepted)  have  on  a  manager’s  acceptance  of  simulation  results? 

***  New  Question  *** 

VERY  STRONG  STRONG  MODERATE  LITTLE  NO  NO  OPINION/ 

EFFECT  EFFECT  EFFECT  EFFECT  EFFECT  DON’T  UNDERSTAND 


C'i  >mment  s  : 
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7.  Given  the  results  of  round  one,  below,  revise  the  rankings  as 

needed . 

To  the  Ief t  of  each  item  is  the  ranking  (1  being  most 
important)  as  to  what  you  feel  has  had  the  most  impact  on  managers’ 
acceptance  of  computer  simulation  results. 

To  the  r ight  of  each  item  is  the  number  of  panel  members 
that  require  personal  knowledge  of  that  factor  when  using  simulation 
results.  Items  that  did  not  require  personal  knowledge  were  assumed 
to  be  of  no  importance  or  were  the  responsibility  of  individuals 
lower  in  the  organizational  structure. 

PERSONAL 


RANK  KNOWLEDGE 

REQUIRED 

__  1  Degree  to  which  model  represents  the  9 

"real  world" 

_  2  Technical  information  incorporated  in  8 

the  computer  model 

_  3  Confidence  in  individual  performing  5 

s imulat ion 

_  4  Confidence  in  individual  presenting  4 

results 

_  5  .Ability  to  assimilate  recommendations  4  _ 

_  6  Previous  experience  with  simulations  3  _ 

_  7  Superior’s  acceptance  simulation  results  5  _ 

_  8  Verification  of  correct  software  coding  1  _ 

_  9  Degree  (magnitude)  of  effort  developing  2  _ 

resul ts 

_  IQ  Cost  of  getting  results  3  _ _ 

_  i 1  Other  program  managers’  acceptance  of  1  _ 

simulation  results 

_  12  Formal  educat ion  courses  in  computer  1  _ 

s imulat ion 

_  13  Tact  used  in  presenting  results  f)  _ 

_  14  Managers’  insights  and  intuition  8 


F'or  the  two  new  factors,  below,  please  indicate  where  within 
the  updated  prioritized  list  of  factors,  above,  they  should  be 
placed. 

A.  "Acceptance  by  user  of  the  simulation’s  ability  to  represent 
the  ’real  world.’" 

Place  between _ and  . 

B.  ”  \eeeptabi 1 i ty  of  the  model  by  the  community  that  generally 
conducts  simulations  in  the  specialty  area.  For  instance- 
there  are  several  ECM  models,  but  none  are  widely  accepted." 

Place  between  and 
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Is  there  anything  else  you  would  like  to  add  or  any  other  factors  you 
feel  should  he  included? 


"Acceptance  by  user  of  the  simulation  representing  the  real  world." 

"Acceptability  of  the  model  by  the  community  that  generally  conducts 
simulations  in  the  specialty  area.  For  instance-there  are  several 
ECM  models,  but  none  are  widely  accepted." 

(NOTE:  Each  of  the  above  listed  comments  were  included  as  new 
questions  in  this  round  of  the  survey.) 


Thank  you  for  completing  this  <^uest ionnai re  and  sharing  your  opinions. 
Please  send  through  distribution  or  mail  this  survey  as  soon  as  possible 
to: 

AFIT/LSG  (Bldg  641)  (Capt  Thomas  Wiggs) 

Wright -Patterson  AFB,  OH  45433-6583 


Appendix  C:  Delphi  Survey  Comments 


1.  What  effect  does  a  manager’s  understanding  of  the  technical 
information  used  in  simulations  have  on  their  acceptance  of 
simulation  results? 

ROUND  1  COMENTS:  "Between  moderate  and  strong  -  too  often  managers 
believe  the  data  is  being  manipulated  to  get  an  answer  if  they  don’t 
understand  the  detail." 

ROUND  2  COMENTS:  NONE 


2.  What  effect  does  the  degree  to  which  a  computer  model  represents  the 
"real  world"  have  on  a  manager’s  acceptance  of  simulation  results? 

ROUND  1  COMENTS:  "Depends:  many  times  we  can  only  guess  at  'real 
world’  but  the  simulation  is  still  valuable  in  choosing  between 
alternatives,  using  the  source  set  of  assumptions,  or  a  controlled 
set  of  variables." 

ROUND  2  COMENTS:  ".Agree  (with  Round  1  comment  -  absolute  numbers 
may  be  off  with  deltas  still  valid!  However,  a  credible  link  to 
reality  must  exist." 

"I  stay  as  before  -  when  the  model  can  represent  the  real  world 
acceptance  is  strong  -  but  there  are  cases  where  we  simply  do  not 
know  the  real  world.  In  this  case,  a  disciplined  approach  to 
modeling  the  real  world  is  still  valuable  to  decision  makers.  Thus, 
the  no  effect  answer  -  useful  for  testing  hypotheses." 

3.  What  effect  does  the  verification  of  correct  software  coding  in  the 
model  used  for  simulation  have  on  a  manager’s  acceptance  of 
simulation  results? 

ROUND  1  COMENTS :  "Think  the  answer  is  little,  but  should  be 
s  t  rong. " 

ROUND  2  COMENTS:  "Incorrect  coding  would  introduce  random  'unknown- 
unknowns’  into  results  and  further  (greatly')  weaken  credibility  in 
resu Its." 

"Agree  with  (Round  1)  comment" 

"Managers  have  little  time  or  even  understanding  of  software 
verification;  thus,  I  say  little  effect.  Any  other  answer  would  be 
speaking  to  the  ideal  rather  than  real  world." 
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4.  What  effect  does  the  simulation  results’  agreement  with  the 
manager’s  insights  and  intuition  have  on  his/her  acceptance  of 
these  results0 

ROUND  1  CONMEXTS :  "Too  often  the  model  results  are  accepted  only  if 
they  support  intuition." 

"Unfortunately,  we  all  suspect  the  model  builder,  because 
simulations  have  been  used  for  political  purposes;  thus,  and  again 
unfortunately,  simulations  that  counter  our  intuition  are  often 
rejected. " 

ROUND  2  COMMENTS:  ".Agree"  (with  Round  1  comment). 

".Agree  with  comment  in  #4.  Tough  to  overcome  inherent  biases." 

"Managers  are  people.  People  have  insights  and  intuition. 

Regardless  of  how  rigorous  the  modeling  effort  or  how  convincing  the 
modelers  might  be,  the  manager  is  still  going  to  rely  to  some  degree 
on  his/her  experience  -  i.e.  intuition  and  insights." 

"I’ll  stay  with  moderate: 

a.  When  I  use  a  model /modeler  with  whom  I  have  experience  and 
trust  -  I  accept  results  that  are  contrary  to  intuition. 

b.  When  I  use  an  unknown  mode 1 /mode  1 er ,  I  question  unintuitive 
results  for  the  reasons  stated  in  your  second  comment." 


o.  What  effect  does  the  general  acceptance  of  simulation  results  by  a 
manager’s  superiors  have  on  his/her  acceptance  of  simulation 
resul t  s? 

ROUND  1  CEMENTS:  NONE 

ROUND  2  CONMENTS :  "If  you  have  to  support  your  decision  up  the 
chain  and  the  boss  does  [not]  accept  simulations,  why  use  them?" 


6.  What  effect  does  the  acceptance  of  simulation  results  by  other 
program  managers  have  on  a  manager’s  acceptance  of  simulation 

results? 


ROUND  1  COMMENTS:  NONE 

ROUND  2  COMMENTS:  "I  changed  because  recently  I’ve  noticed  more 
managers  being  swayed  to  some  degree  by  others’  opinions." 

"If  we  were  all  talking  about  the  same  model  find  the  same  modeler,  I 
might  move  to  strong.  Otherwise,  I  don’t  care  what  other  FM  think 
of  simulations.  They  are  valuable  to  see." 
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7.  What  effect  does  the  degree  (magnitude)  of  effort  used  in  developing 
the  simulation  have  on  a  manager’s  acceptance  of  the  results? 

ROUND  1  COMMENTS :  NONE 

ROUND  2  OOWENTS:  "Let’s  be  real,  as  a  PM  I  don’t  care  how  hard  it 
is  to  do  a  simulation.  If  the  modeler  can  do  it  and  the  simulation 
is  valuable  to  the  decision  process,  we  go  for  it.  If  it’s  not 
valuable  to  the  decision  we  don’t  do  it  no  matter  what  the  effort 
is." 


8.  What  effect  does  the  manager’s  ability  to  understand  recommendations 
resulting  from  simulations  have  on  his/her  acceptance  of  the 
simulation  results? 

ROUND  1  OCWENTS:  NONE 

ROUND  2  COMMENTS:  NONE 


0.  What  effect  does  a  manager’s  confidence  in  the  individual  presenting 
the  results  have  on  his/her  acceptance  of  simulation  results? 

ROUND  1  COt^MENTS:  "The  simulator  better  be  able  to  defend  every 
variable  and  assumption.  Consistency  and  practicality  are  what  I 
look  for.  Did  he  change  his  model  for  this  simulation?  If  so  why?" 

ROUND  2  COMENTS:  "Agree"  (with  Round  1  comment). 


10.  What  effect  does  confidence  in  the  individual  performing  the 

simulation  have  on  a  manager’s  acceptance  of  simulation  results? 

ROUND  1  CONMENTS  :  "Cons  istency/Honesty" 

ROUND  2  COMENTS:  "Quite  often,  the  manager  has  no  idea  who  is 
running  a  s imulat ion . " 


11.  What  effect  does  the  cost  of  obtaining  the  simulation  have  on  a 
manager’s  acceptance  of  simulation  results? 

ROUND  1  COMMENTS:  NONE 

ROUND  2  CT>WENTS:  "A  credible  presentation,  we  1 1 -prepared  and 
skillfully  presented,  whether  it’s  simulation  results  or  something 
else,  can  strongly  affect  a  manager’s  acceptance." 

"Cost  is  a  decision  I  make  before  I  start.  Once  I  have  the  results 
in  hand,  it  is  too  late  to  make  cost  influence  acceptance." 
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12.  What  effect  does  a  manager’s  previous  experience  with  simulation 
have  on  their  acceptance  of  simulation  results  later? 

ROUND  1  COMENTS:  NONE 

ROUND  2  OOMENTS:  NONE 


13.  What  effect  does  skill  used  in  presenting  the  simulation  results 
have  on  a  manager’s  acceptance  of  these  results? 

ROUND  1  CX*MENTS:  NONE 

ROUND  2  CONMENTS:  "The  PM  has  got  to  understand  what  he  is  being 
told.  If  the  presentation  does  not  contribute  to  understanding  how 
can  there  be  acceptance?" 

14.  What  effect  does  formal  education  courses  in  computer  simulation 
have  on  a  manager’s  acceptance  of  simulation  results? 

ROUND  1  (XMMENTS:  NONE 

ROUND  2  CONMENTS :  "I  was  strong  because  I  believe  the  more  you  know 
about  the  capabi 1 i t ies/1 imi tat  ions  of  simulation  as  a  management 
tool  the  better  you  can  use  them;  but  i’ve  changed  because  I  know 
PM’s  with  no  education  who  use  them  well." 


15.  What  effect  does  the  acceptance  by  the  user  of  the  simulation’s 

ability  to  represent  the  "real  world"  have  on  a  manager’s  acceptance 
of  simulation  results? 

***  New  Question  on  Round  2*** 

ROUND  2  COMMENTS:  NONE 


16.  What  effect  does  the  acceptability  of  the  model  by  the  community 
that  generally  conducts  simulations  in  the  specialty  area  (for 
instance--  there  are  several  ECM  models,  but  none  are  widely 
accepted)  have  on  a  manager’s  acceptance  of  simulation  results? 


***  New  Question  on  Round  2*** 


ROUND  2  COMENTS:  "Utility  and  integrity  are 
community  has  rejected  a  model  they  must  know 
models  are  tools  -  not  the  answer.  Why  use  a 


important.  If  the 
something.  Remember 
defective  tool?" 
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Is  there  anything  else  you  vould  like  to  add  or  any  other  factors  you 
feel  should  be  included? 


ROUND  1  COMENTS :  "Acceptance  by  user  of  the  simulation  representing 
the  real  world." 

"Acceptability  of  the  model  by  the  cr mmuni ty  that  generally  conducts 
simulations  in  the  specialty  area.  For  instance- there  are  several 
ECM  models,  but  none  are  widely  accepted." 


ROUND  2  COMENTS:  "Track  record  of  individual  or  organization 
performing  the  simulation." 

"May  already  be  included  but  it  seems  that  official  sanction  by 
competent  authorities  designating  the  use  of  a  simulation  model  for 
certain  purposes  enhances  its  acceptance  in  the  larger  community. 
For  instance,  I  am  thinking  of  the  Logistics  Composite  Model  (LCCM) 
which  the  Air  Force  Manpower  Community  has  validated  as  the 
picfci-ied  model  to  conduct  manpower  analyses.  As  such  it  has 
maintained  its  credibility  over  time  despite  some  obvious  short 
comings . " 

"I  am  big  on  understanding  the  assumptions  used  before  the 
simulation  is  run.  Playing  with  assumptions  can  cause  wide  swings 
in  the  results;  thus,  if  I  disagree  with  or  don’t  understand  an 
assumption,  seeing  the  simulation’s  sensitivity  to  that  assumption 
often  increases  my  acceptance  of  the  results." 


64 


Bibliography 


1 .  Amo  Id,  W  i  1  bur  V .  Designing  Defense  Systems ,  The  Trade-off 
Process .  Monograph,  DSMC-SE-T.  Defense  Systems  Management  College 
(DSMC),  Fort  Belvoir  VA,  November  1986  (AD-A192997) 

2.  Beardsley,  Tim  "Star  Games,"  Sc ient i f ic  American,  258 :  18-22 
(May  1988) 

3.  Biles,  William  E.  "Introduction  to  Simulation,"  Proceeding's  of  the 
1987  Winter  Simulation  Conference ,  edited  by  A.  Thesen,  H.  Grant 
and  W.  David  Kelton.  7-15.  Atlanta  GA,  1987. 

-1.  Brovn,  Bernice  B.  Delphi  Process :  A  Methodology  Csed  for  the 
Elicitation  of  Opinions  of  Experts .  Santa  Monica  CA:  Rand 
Corporation,  September  1968  (AD-675981). 

5.  Dalkey,  Norman  C.  Delphi.  Santa  Monica  CA:  Rand  Corporation, 

October  1967  (AD-669554). 

6 .  - .  The  Delphi  Method:  .An  Experimental  Study  of  Group 

Operation .  Contract  F44629-67-C-9945 .  Santa  Monica  CA:  Rand 
Corporation,  June  1969  (AD-699498). 

7.  Dumond,  Lt  Col  John,  Head,  Department  of  System  Acquisit ion. 
Personal  intervievs.  Air  Force  Institute  of  Technology,  School  of 
Systems  and  Logistics.  Wright -Pat ter son  AFB  OH,  September  1988 
through  June  1989. 

8.  Emory,  C.  William  Bus iness  Research  Methods .  Homevood  Illinois: 
Richard  D.  Irvin,  Inc.,  1985 

9.  Glazer,  Rashi  and  others  "Group  Process  and  Decision  Performance 
in  a  Simulated  Marketing  Environment,"  Journal  of  Business 
Research,  15 :  545-557  (December  1987) 

19.  Gregor,  Capt  Ralinda  B.  A  Normal ive  Mode 1  of  the  Ideal  Qual i t ies , 
Characteristics ,  and  Background  of  the  Senior  Air  Force  Civi 1 ian 
Logistician.  MS  Thesis,  AFIT/GLM/LSM/88S-27 .  School  of  Systems 
and  Logistics,  Air  Force  Institute  of  Technology  (AU) ,  Wright  - 
Patterson  AFB  OH,  September  1988 

11.  Jones,  Harry  and  Brian  C.  Tviss.  Forecast ing  Techno I ogy  f or 
Planniiig  Decisions .  New  York:  The  Macmi 1 1 ian  Press  Ltd.,  1978 

12.  Keating,  Barry  "Simulations:  Put  the  Real  World  in  your  Computer," 

Creative  Comput ing,  1 1 :  56-64  (November  1985) 

13.  Martino,  J.  Technological  Forecasting  for  Dec i s i on  Making. 

Nev  York:  Elsevier  Publishing  Company,  1983. 


65 


14.  Morris,  William  T.  Implementation  Strategies  for  Industrial 
Engineers.  Columbus  Ohio:  Grid  Publishing  Inc.,  1979. 

15.  Nash,  Colleen  A.  "The  Twenty-two  Most  Critical  Technologies,"  Air 
Force  Magazine,  72:  25  (June  1989). 

16.  Neelamkavil,  Francis.  Computer  Simulation  and  Model  I  ing.  New 
York:  John  Wiley  and  Sons,  1987. 

17.  Ot t ,  Lyman .  An  Introduction  to  Stat ist ical  Methods  and  Data 
.Analysis ■  Boston:  PWS-KENT  Publishing  Company,  1988 

18.  Pidd,  Micheal.  Computer  Simulation  in  Management  Science.  Great 
Britian:  John  Wiley  and  Sons  Ltd.,  1984. 

19.  Pritsker,  Alan  B.  Introduct ion  to  Simulat ion  and  SLAM  II .  West 
Lafayette  Indiana:  Systems  Publishing  Corporation,  1986. 

20.  Sargent,  Robert  G.  "Am  Overview  of  Verification  and  Validation  of 
Simulation  Models,"  Proceedings  of  the  1987  Winter  Simulat ion 
Conference ,  edited  by  A.  Thesen,  H.  Grant  and  Wr.  David  Kelton. 
33-39.  Atlanta  GA,  1987. 

21.  Stanislaw,  Harold.  "Tests  of  Computer  Simulation  Validation," 

Simulation  And  Games ,  17 :  173-191  (June  1986) 

22.  T  .scari,  Major  Thomas.  Class  lecture  in  SMGT  642,  Research  and 
Development  Management.  School  of  Systems  and  Logistics,  Air  Force 
Institute  of  Technology  (AU) ,  Wright -Patterson  AFB  OH,  31  January 
1989. 


66 


VITA 


Contain  TVinmAs  Kent  Wicftrs 


He  then  attended  Virginia  Military  Institute  (VMI) 


from  which  he  received  a  Bachelor  of  Science  degree  in  Electrical 
Engineering  in  May  1981. 

Upon  graduation,  he  received  a  commission  in  the  USAF  through  the 
ROTC  program.  Captain  Wiggs  reported  for  initial  duty  on  June  27,  1981 
at  Offutt  AFB,  Nebraska  assigned  to  the  544th  Strategic  Intel legence 
Wing  (SIW)  as  a  Minuteman  Til  Ballistic  Missile  Trajectory  mgineer.  He 
then  served  as  Chief,  Advanced  Cruise  Missile  Ground-Based  Threat  Test 
at  Edvards  AFB,  California.  In  1988  Captain  Wiggs  entered  the  Air  Force 
Institute  of  Technology,  School  of  Systems  and  Logistics,  as  a  graduate 
student  in  Systems  Management. 


UNCLASSIFIED 


SECURITY  CLASSIFICATION  OF  This  »AGE 


la  REPORT  SECURITY  CLASSIFICATION 

UNCLASSIFIED 


2a.  SECURITY  CLASSIFICATION  AUTHORITY 


2b.  DECLASSIFICATION /DOWNGRADING  SCHEDULE 


4.  PERFORMING  ORGANIZATION  REPORT  NUMBER(S) 

AFIT/GSM/LSY/89S-4 4 


REPORT  DOCUMENTATION  PAGE 


lb.  RESTRICTIVE  MARKINGS 


Form  Approved 
OMB  NO.  0704-0188 


3  .  DISTRIBUTION /AVAILABILITY  OF  REPORT 

Approved  for  public  release; 
distribution  unlimited 


5.  MONITORING  ORGANIZATION  REPORT  NUMBER(S) 


6a.  NAME  OF  PERFORMING  ORGANIZATION  6b.  OFFICE  SYMBOL  7a.  NAME  OF  MONITORING  ORGANIZATION 
School  of  Systems  and  (if  applicable) 

Logistics  AFIT/LSY 


6c  ADDRESS  (City,  State,  and  ZIP  Code) 

Air  Force  Institute  of  Technology 
Wr: ght-Patterson  AFB,  OH  45433-6583 


7b.  ADDRESS  (City,  State,  and  ZIP  Code) 


8a.  NAME  OF  FUNDING /SPONSORING 
ORGANIZATION 


8b.  OFFICE  SYMBOL 
(If  applicable) 


9  PROCUREMENT  INSTRUMENT  IDENTIFICATION  NUMBER 


8c.  ADDRESS  (City,  State,  and  ZIP  Code) 

10.  SOURCE  OF  FUNDING  NUMBERS 

program 

PROJECT 

TASK 

WORK  UNIT 

ELEMENT  NO 

NO. 

NO 

ACCESSION  NO. 

DETERMINATTCN  OF  THE  FACTORS  AFFECTING  THE  ACCEPTANCE  OF  COMPUTER  SIMULATION  RESULTS 


12.  PERSONAL  AUTHOR(S) 

Thomas  K.  Wiggs,  B.S.,  Capt,  USAF 


13a.  TYPE  OF  REPORT  13b.  TIME  COVEREO 

MS  Thesis _  from _ to 


16.  supplementary  notation 


14.  DATE  OF  REPORT  (Year,  Month,  Day)  15.  PAGE  COUNT 

1989  Septarber  79 


COSATI  CODES 


GROUP 


,18.  SUBJECT  TERMS  ( Continue  on  reverse  if  necessary  and  identify  by  block  number) 

sub-group  | -Computerized  Simulation;  Air  Force  Procuranent; 

Decision  Making;  Delphi  Techniques  -  — v  ••• 


‘9.  ABSTRACT  ( Continue  on  reverse  if  necessary  and  identify  by  block  number) 

Thesis  Advisor:  John  Dumond,  Lt  Col  USAF 

Head,  Department  of  System 
Acquisition  Management 
School  of  Systems  and  Logistics 


Approved  for  public  release:  I  AW  AFR  190-1. 

0/. 

LARRY  W.  EMMELHAINZ,  Lt  061,  USAF  14  Oct  89 
Director  of  Research  and  Consultation 
Air  Force  Institute  of  Technology  (AU) 
Wrlght-Patterson  AFB  OH  45433-6583 


20  distribution/ availability  of  abstract 
guNCLASSlFlEQ/UNLIMlTED  □  SAME  AS  RPT  □  D- 


22a.  NAME  OF  RESPONSIBLE  INDIVIDUAL 

John  Dumond,  Lt  Col,  USAF 


DO  Form  1473,  JUN  86  Previous  e< 


21  ABSTRACT  SECURITY  CLASSIFICATION 

□  otic  users  UNCLASSIFIED 


22 b.  TELEPHONE  (Include  Area  Code)  22 c.  OFFICE  SYMBOL 

(513)  255-3355 


Previous  editions  are  obsolete.  SECURITY  CLASSIFICATION  OF  THIS  PAGE 

UNCLASSIFIED 


UNCLASSIFIED 


For  the  Air  Force  to  continue  developing  and  acquiring  today’s 
highly  complex  and  costly  veapon  systems,  it  will  have  to  depend  more 
heavily  on  computer  simulations  to  fill  in  where  hardware  is  not  avail¬ 
able  and  actual  testing  cannot  be  performed.  The  usefulness  of  these 
computer  simulations  are  limited,  how-ever,  by  the  degree  to  which  the 
results  are  accepted  by  managers.  From  a  review  of  the  literature, 
how-ever,  it  would  appear  that  no  previous  research  has  been  performed  to 
determine  Air  Force  program  managers’  perceptions  of  the  acceptability 
of  computer  simulation  results.  In  light  of  that  finding,  the  two 
objectives  of  this  thesis  were  to  identify  the  factors  affecting  the 
acceptance  of  computer  simulation  results  by  senior  program  mangers 
within  AFSC/ASD  and  to  determine  the  relative  importance  of  the  identi¬ 
fied  factors. 

/ 

Because  this  research  was  exploratory  in  nature,  the  Delphi  data 
collection  technique  was  used.  An  expert  panel  of  nine  Air  Force 
colonels  completed  the  Delphi  survey.  The  experts  arrived  at  a  list  of 
sixteen  factors  that  affect  the  acceptance  of  simulation  results.  Of 
these  sixteen  factors,  only  ten  were  said  to  have  a  "strong  effect." 

The  conclusions  drawn  from  the  expert  panel  responses  show  that 
senior  program  managers  are  principally  concerned  about  the  simulation 
representing  the  "real  world"  and  about  the  technical  information  incor¬ 
porated  into  the  model.  The  experts  also  expressed  that  the  manager 
needs  to  know'  if  the  end  user  of  the  weapon  system  believes  the  simula¬ 
tion  to  be  accurate.  Several  of  the  other  factors  included:  confidence 
in  the  individual  performing  the  simulation,  ability  to  assimilate  any 
recommendations,  acceptability  of  the  model  by  the  community  that  gener¬ 
ally  conducts  simulations  in  the  specialty  area,  confidence  in  the 
individual  presenting  the  results,  previous  experience  with  simulations , 
superior’s  acceptance  of  simulation  results,  and  managers’  own  insights 
and  intuition. 


UNCLASSIFIED 


